Layered Architecture of BubbleBoy

            Layered architecture ( figure) of BubbleBoy is a detailed interpretation of main components of the system. Layers of the systems designed as tree tier architecture. Main tears can be identified as Presentation Tier, Logic Tier and Data Tier. It will clarify the system functionality in abstract and it will clear the main layers and sub layers of them.

Logic Tier

This layer concludes the software application part of the BubbleBoy.

·         Image Acquisition

            Preparing of the images taken from user and feeding them to image processing will be done at this sub layer.

·         Image Processing

            This layer consists of three parts 2D image processor, 3D Image generator, Head Direction capture module. Initially the image is in the position came from the image acquisition sub layer and it will contain the marker with it.

·         Gesture Recognition

            This represent the recognition of the pose of the user in one time instant and the details will be compared with the knowledge base data. Then the results will be proceeds to the action detection layer.

·         Action Detection

            In this layer it will map the sequence of the images processed by the above layer with the details on human actions in Knowledge base and will detect the action of the user.

·         Knowledge Base Access Handling

            This layer is responsible for handling the communication of Knowledge Base with Logic Tear components. It will control the access requests from application and manage the data flow of the in and out of Knowledge Base.
Environment Generation

            This layer will take inputs from the action detection layer; generate appropriate environment change according to detected action of the user.

·         Environment Modification

            Environment Modification layer is the layer that will make changes immediately after the changes in environment are identified by the Environment generation layer.

Data Tier

·         BubbleBoy Database

            Database will be required for the Logic Tier components to store data apart from Knowledge Base. The Knowledge Base will be keeping specialized data for gesture recognition and action detection. Therefore the database for the other components of the Logic Tier will be necessary. 

·         Knowledge Base

            Knowledge Base will consist of two types of data basically. Gesture model details and Action detection details can be identified in this stage as details to be included in Knowledge Base.

 Presentation Tier

·         Graphical Environment

            This will be showing the virtual environment to the user according to the action he/she presented.

Four Major Phases of BubbleBoy

The system BubbleBoy can be divided in to four major phases in its development process. They are,

                               I.            Human Computer Interaction phase

                            II.            Image Analysis Phase

                         III.            Action Recognition and

                         IV.            Environment

Phase   I

            This step is the initiation step of the complete operation. It will capture images and feed them to the phase II. It will capture the image with passive markers on user. Two cameras will be used to the capturing operation.

Phase II

            Initially the image is in the position came from the first phase of the system and it will contain the marker with it. It will progress in to next step. Then it will generate limb recognition with the help of the markers. The most critical part of this phase is in the next step. The 3D model will be generated using limbs recognized. Angles and the sizes will be main information for the conversion.

Phase III

            Since this need to be implemented for a series of images which are fed by video camera, it need to be evaluated all incoming images efficiently. So the series of generated 3D models will be supplied to the next phase. This phase’s main task is to match the action according to the trained set of actions.

Phase IV

            This will be the user feedback phase as the user can watch the results of the action he/she presented.

Project meeting minute - 09

Date & time, duration

The first meeting was held on 2010- 12 -02 for one hour

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Feedback on the second iteration of the Software Architecture Document (SAD)

Brief Discussion of the discussion

Today also we basically discussed about the content of Software Architecture Document. This time we presented our second iteration of the document and sir explained us the deficiencies we have made there. Those are

• Maintain the consistency of the document control

• Come up with diagrams that looks fine

• Use correct terms such as gesture recognition, image acquisition and action detection.

• Design document consisting of two components such Final design of our proposed system § as Assurance of our design with the work carried by other people(taken § from trusted sources)

• In the image acquisition phase sometimes we have to exclude some of the gestures depending on the context where we will be at that particular moment (attention on gesture inclusion and gesture exclusion)

• Explain the design for the 3-D model creator and its state of art(While we have design for overall system we have to have designs for all the sub components come under the overall system)

• And for every design it should carry the rationale behind it, historical work, current trends and future directions.

• And also we were asked to look at the web resource “http://www.cs.sfu.ca/~mori/research/”

Key topics discussed

1. Compare the things we did with other people in the history who has contributed to the things we are referring to.

2. Maintain the standard of the referencing process.

3. Maintain a log book for the things we carry out with related our project

Comments and agreements

Chandana sir made us understand the important points that we have to improve and asked those corrections to be made in the next iteration of the document and send him that version of the document tomorrow. And since the references are not in standard way once they are created from different ways sir suggested us to go to the latex which uses an approach called bib Text to have references in standard way.

1 Design Doc Introduction

In the present emergent market, it is very trivial to locate and experience new advancement of the Human Computer Interaction but with some constrains. This is a field which still not unveiled to the society but could talk to a majority of population who do not have enough computer literacy at the moment. This emergent innovative field is still at research level at a certain degree. At a time where there are rapid advances in the field of Human Computer Interaction we have decided to give it another interpretation through the fulfillment of our project

The project “BubbleBoy” is an innovative, interactive 3D interface using full body motion. Obviously the interaction between the human and computers is done at the user interface which includes both software and hardware. Sometimes one may think is this something to be invented as there are some applications already of same nature and in that case is there a room for a research component.

HCI was one of the first examples of cognitive engineering, an approach which crosses boundary between academic disciplines and modern needs and professions have emerged; to effective human interaction. At the same time with the emergence of the area Software Engineering more attention was focused on non functional requirements such as usability and maintainability.

During the project BubbleBoy, it is required to follow certain software development model in order to achieve better quality output. Since the requirements and technologies which are changing rapidly, it is preferred to use a dynamic model which well comes any deviation without a conflict. In this project, it can identify a project life cycle with five different phases. Those are identification, plan, implementation, testing and completion. As it is planned, new technologies are welcome during any phase. So this eventually becomes highly flexible and up to date implementation. Even new ideas could arise while doing implementation. To create more competitive product, it should be up to date. This project basically can be defined as identification of an opportunity in the market. To meet up with these requirements, it is preferred to use an evolutionary/iterative model as software development model and at the same time team would make sure to adhere with coding paradigms and standards.

1.1 BubbleBoy the Innovative Interface

BubbleBoy is a project which facilitates the presence of hundreds of public who do not have enough computer literacy. This interface consists of several components which makes the life easy for the user.

• Environment which talk back to the user

• Reduced Processing cost due to Reduction in image processing.

• Reduced sophisticated hardware parts and do all those tasks in software phase

• Growing Artificial intelligence facilitates higher flexibility

Project consists of a sound environment which gives the experience of the real environment. This project will voyage the user to believe virtual realistic and demonstrate high interactive level with virtual world. To make user feel immersive, environment around user need to talk back to him as real environment do. As well as environment should consist of several continuous and distinct activities. Consider a user is walking through a jungle and stops where a leaf of a banana tree ahead. Now his vision is limited by that leaf. Now user is trying to lift the leaf and look forward. Then with a particular he must able to move it up ward and have the view ahead precisely. After he releases the leaf, it should come opposite direction and have some king of fluctuation until it become stable. That’s the level of talk back is expected from the environment.

This project uses makers to identify the movements rather than using pure image processing techniques. That was done with a great visionary. Image processing and identify user movements also possible. But it consumes more resources and utilizes much CPUs` processing power. This would be a higher risk since this is modeled by targeting the public market where there is lack of computers with enormous processing power. Not only that, this could possess the team to deviate from the core activities of the project.

One of the most important fact relate with this project is the independency of sophisticated hardware. As we can see there are few products on the current market, but with expensive hardware. This project will work with the aid of cheep web cameras which are affordable to any user.

This project will consist of Artificial Intelligence which would be able to grow with the experience. This component is going to use in user action recognition area and this will facilitate a system which is getting matured with time elapse.

1.2 BubbleBoy Implementation and Integration

Implementation of this project has three main components which should work independently. This independency facilitates the implementer to work without any conflict with other components. The entire process related to the project could be demonstrates by the Figure 1 and those components could be classified as follows.

• 2D to 3D modeling using maker based protocol

• Artificial Intelligence

• Environment

In 2d to 3d component it considers the image captured from the user in a sequential manner. This capturing tries to identify the markers on the user and develop a model from those details. Initially two web cameras are capturing images from the user and system which is going to implement would consider the time base and merge image to create a 3D model of the user. This process will continue for sequence of images and generate a sequence of 3D models. And these models will be forwarded to the next stage.

Artificial Intelligence system will be developed as a separate component. This component will get 3D model sequence as the input and deliver a prediction as the output which would be compatible with the inputs of the environment. This unit will examine 3D model sequence in precise manner and try to predict the user movement. Meanwhile this unit has the ability of growing with the experience. This will sharpen the probability of correct prediction.

The virtual environment will act as the third and one of the important parts of this project. This unit is responsible to make the user feel the virtual realistic. This will be consist component such as environment talk back, sequential activities and distinct activities. This is the interface which user interacts with. So this should be well defined system in order to preserve the quality of entire project. Inputs will be provided from the AI and output is the experience get by the user. According to the user input environment must be able to change some artifacts of the virtual world in a real world manner.

1.3 Summary

This particular chapter gives a brief introduction of the project Bubble Boy which is innovative HCI. It is major discussed higher overview of the project in different perspectives while looking at appropriate project life which is suitable for entire project. Then under the sub section BubbleBoy the innovative interface, it describes more into the depth by describing several features which make the life easy for the user. The section of BubbleBoy Implementation and Integration, deals with the implementation aspects of different components and integration of those components without making constrains on integration.

Project meeting minute - 08

Date & time, duration

The first meeting was held on 2010- 11 -25 for about 30 minutes.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Regarding writing the Software Architecture Document (SAD)

Brief Discussion of the discussion

Today we basically discuss the topic of what should be the content that should be included in the project Architecture document. o Use of phrases like “To make the user feel immersive ” o Need for the components to be independent in the implementation process o Separate assumptions from the dependencies User will not be equipped any kind of dynamic § o Under assumptions High § Number of cameras § active devices but the passive markers resolution cameras will not be used Single user interaction § o Not to use the concept of 360 angle o How the environment get changed based on a fixed reference(point out these things as the facts which accounts this application to become unique among others) o Need behind the components to be independent in the implementation process o The high level architecture which shows the flow of the information o Architecture consisting of two main levels

• Information flow level (User to environment, Environment to user, And within the environment)

• Functional level o Interaction cues received by the application o Information flow towards the user (Statically determined information, dynamically determined information) o All what Chandana sir want is to get us to think about the proceedings of the project with our own imaginative thoughts

Key topics discussed

1. Document writing style guidelines

2. Feedback on the draft SAD.

Comments and agreements

Chandana sir asked us to come up with draft software Architecture Document in the next day and do the researches on key topic areas he has already mentioned.

Project meeting minute - 07

Date & time, duration

The meeting was held on 2010-11-19

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Present health of the project and design document

Brief Discussion of the discussion

This is the second meeting for this week and what we discussed today was to what is the present situation of the project what it should be like in future. Here Chandana sir explained us some important points that should be illustrated within the content of Design document. Following are some of them.

• In the scope of our project how we are planning to attach the user in the virtual environment in physical sense compared to the other system in which the user is detached.

• How we are going to give the immersive feeling to the user in a constrained environment such as a forest, park or a aqua world

• What are the actions a user can do based on that constrained environment he is in.

• Within our system how environment talks to the user with or without a user talks to the environment while in other systems environment talks only when user talks to the environment And for the things we have to carry in advance he asked us to

• Define set of environments and think on how to construct those environments

• Think on how and when to talk to the user by the environment Chandana sir also said in answering to the question raised by Thusitha that since this is a real time system there may be things process level parallelism and system level parallelism(modules working concurrently) for sure from the technical aspect to achieve the targets of the proposed system. The very important fact that Chandana sir mentioned was to read trustworthy kind of research papers so that we can improve the writing things in scientific style and learn the previous work people have done regarding those areas of our interest. While reading those research papers we have to abstract the varieties approaches they have followed and check whether those can be applied to our project based on the requirements. Among those range of varieties of approaches sometimes we can use them as it is or sometimes with some modifications being added.

Key topics discussed

1. Contribution of the each project group member

2. Content that should go into the design document

Comments and agreements

1. Send Chandana sir a draft design document

2. Design the system targeting single user system rather than multiple users system

3. Keep a master plan for the things we do

4. Come with a record of the references we have gone through with the rationale behind them and a summary of that papers when we are coming to the next meeting.

Project meeting minute - 06

Date & time, duration

The meeting was held on 2010-11-18

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

1. Structure for the Software design document

2. Present health of the project

Brief Discussion of the discussion

Today basically gave his feedback on the draft structure of the Design document we have sent to him through mail. He explained what the things that should go to each sections of the design document. And more importantly he praised us for doing great job in order to come up with proper SRS. Anyway we wouldn’t be able to do that without his guidance was not there. So the regarding the section of “Introduction” of the design document he explained following are the things that should go into.

• Several passages on the nature of the project

• Why the exploratory development model as the software process model

• Within our project how we are moving from well structured and well established domain which includes Image analysis, HCI etc into a not well established area of implementation As Chandana sir explained in the section of methodology we have to explain the things like

• Type of the interactive system.

• The nature of the exploratory work we have to carry because of the inherent innovative work included with the project Other than that we have to explain the following also

• The rationale behind choosing exploratory development model with related to the requirements and future technologies to come

• How we are going to present the virtual experience to the user as he feel like real without being in the place.

• How we present the system to the user so that he will readily pay something to be of that.

• How to immerse the user in the virtual environment

• How far we can go with visual perceptions And regarding the format of the design document Chandana sir told we can use the Rup templates as the base and not to depend on that structure.

Key topics discussed

Structure and the content of the Design document

Comments and agreements

Chandana sir asked us to come up with draft design document and meet him on tomorrow.

High-level System Overview

This projects main target is to provide a product to interact with the computer without wearing any advance equipment on the human body. High level architecture of this project would explain briefly in this section. Web cam would act as the primary input device for the project while spots on the user act as identification. User will wear spots on him and act his role to interact with the computer. These images are captured with the simple web cameras as a video stream. It could use 15, 30 or higher frames per second but it depends on the processing power of the machine. Images on this video streams will capture. Then a unique algorithm will capture the human image along from the frame. then It considering the image sequence captured and try to model it in to a 3D model and predict the movement of the user. This process would consume lot of processing power in this project. Identifying the movements of the user would be done through an artificial neural network. This can grow the knowledge base as it gets matured with time. With the aid of this neural network the engine would give the information of the user movement to the environment. Then user should be provided with a high end graphical user interfaces to interact with. This project contains environment that user could interact with and have a real life experience. According to the input from the prediction the environment would change. It could be certain objects in the environment or the entire frame [34].
 Usability
This projects` main target is to deliver a simple and user friendly interface to interact with the computer.
o Training Time
It is anticipated to deliver an output which would makes easy to the user, to be familiar with the system. It would be the case in both normal user and power user.
o Switching Cost
User will able to switch between activities without any issue or cost. Since the all the controlling aspects of the system would control entire by user movement.
 Reliability
o Accuracy
It is required to deliver a high resolution environment with product. Accuracy acts a main role in this scenario. This has to demonstrate a high accuracy when changing objects and frames of the environment. Each activity has to be continuous, as well as well defined.
o Mean Time to Repair
The mean time to repair would be lower, since the best practices and standards are used with project implementation. Using well defined design patterns in coding would help on this.
 Interfaces
o User Interfaces
According to the user movements the system will capture the frames and process. And then the environment should change according to the movement of the user. To estimate the motion, it is require considering overall body movement. And respect to the motion it uses a neural network related thing and try to identify the motion. It will identify as the movement as related to the natural human movements. Then it would consider the real condition and the nature of the current environment. In the natural case, area of the scene changes only when eyes move. Other than that it should be the same angle of the environment. But objects of that area could affect by the user movement.
o Hardware Interfaces
One of the main requirements of this project is to deliver an innovative computer interactive method without using any additional hardware. We only use cheep web cameras to capture the images.

o Software Interfaces
Designing a Graphical Environment was done related to the project, while going through the research. This projects` main target is to interact with the movement of the user in real time. As the project requirement, this interaction should reflect via a high graphical interface. So it was tested at several choices and did some testing to ensure the ability of available resources [18, 19].

Project meeting minute - 05

Date & time, duration

The meeting was held on 2010-10-29.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Contribution to the project by each member

Brief Discussion of the discussion

As the deadline of the SRS was scheduled to be on 5th of November we discussed the important facts on that and Chandana sir wanted every of us to come up with the contribution for the project during that week. So we told to the sir what we have done on particular areas of the project since we have divided those topics among ourselves beforehand. Lahiru the team leader of our project had come up with his findings on the topic HMM (Hidden Markov Model) along with other details related to the area 2D to 3D mapping. In my case I have done my research on neural networks as a replacement methodology for HMM. And Thusitha and Hasitha did their researches on markerless models and interpretation of 3D virtual environment respectively. Chandana sir appreciated our effort and encourage us do further research on selected areas as we have finalized. For an instance for modeling 3D objects out of 2D model he advised us use neural networks rather than HMM based on the fact of the learning ability of neural networks. More importantly apart from the project related works he asked us to study other subjects also and ask us to study other subjects well also and not to let down them because of the project.

Key topics discussed

1. HMM

2. The good side of artificial neural networks over HMM

3. Approach of marker based models

4. Interpretation of 3D environments

Comments and agreements

Today sir went through the things we did as according to the topics we were assigned and make his feedback on several design decisions and asked us to continue those topics further ahead.

Features and operating environment of BubbleBoy.

BubbleBoy Features

BubbleBoy is a project that targets creating a simple user interface for the user to communicate with computer.  It contains following features,

• Operates with commodities like web camera which is easy to find for the user.
• Achieve all the hard work of program using software and user need not to be troubled about installing difficult hardware.
• Track the full body motion of the user and simulate the computer program accouding to it.
• Provides different kinds of experience (different environments) to user as it helps different users’ choices.   
• Provides a less complicated and low cost product to have an invaluable experience.

Operating Environment of BubbleBoy

BubbleBoy provides a scenery of a natural environment experience through computer screen.And it gives the chance of interaction with computer using full body motion(Actions). Operating environment of the system includes following features.

• Two web cameras for capturing the actions of the user.
• Computer with
• High processing power: Core 2 Duo 2.2 or more
• Secondary storage of 5GB separated for the application.
• System memory of 1GB or more

Sceneries that will be there for the end users

In the project "BubbleBoy"we are hoping to deliver a system which would enable interaction without any specific hardware development, but only with aid of software. Only thing user has to do is to paste the markers on him or her which another technique we have exploited to provide the user with better performance and effective outcome. We need to provide appropriate scene as required by the user. The user will enable to select one of following activities.

· Swimming in an aquarium/sea/river.

· Walking alone by seaside/riverside.

· Walking at a museum

· Drive a racing car

· Walking on moon/space

· Having a walk in a volcanic site

· Tsunami

· Surfing

· Snow board

We can add any kind of scenery since we are hoping to capture the whole body movement. We are hoping to design the graphical environment similar to natural conditions to some extent.

It should be noted that in the process of proving scenery to this application we have focus our attention on physically impaired people who are unable of having these kinds of experience during their real life the apart from the average users. They will definitely enjoy it as our application will be fully enriched with life like experiences.

Sofware Specification

When we go through the research we look at the environment design related to the project. This projects` main target is to interact with the movement of the user in real time. As the project requirement, this interaction should reflect via a high graphical interface. So we look at several choices and did some testing to ensure the ability of available resources.
We consider following choices
• XNA
• Crytec
• JMonkey
XNA
XNA is a framework is developed based on native implementation of .NET Framework 2.0 on Windows. The framework runs on a version of the Common Language Runtime that is optimized for gaming to provide a managed execution environment. The runtime is available for Windows XP, Windows Vista, Windows 7, and Xbox 360. Since XNA games are written for the runtime, they can run on any platform that supports the XNA Framework with minimal or no modification. Games that run on the framework can technically be written in any .NET-compliant language, but only C# in XNA Game Studio Express IDE and all versions of Visual Studio 2008 are officially supported.
JMonkey
jMonkey Engine (jME) is a high-performance 3D game framework, written entirely in Java. OpenGL is supported via LWJGL, with JOGL support in development. For sound, OpenAL is supported. Input via the keyboard, mouse, and other controllers is also supported. The most important thing in this is, it run on Windows, Mac OS, and Linux; true cross-platform thanks to the JavaVM.
Crytect
CryEngine is a high performance 3D game engine developed by crytec and they recently release CryEngine 3 . The new engine is being developed for use on Microsoft Windows, PlayStation 3 and Xbox 360. As for the PC platform, the engine is said to support development in DirectX 9, 10, and 11. It supports and contains rich graphical usages such as water reflection, Flow graph, Real time soft particle system & integrated FX editor, Road & river tools, Vehicle creator, Real time dynamic global illumination, Deferred lighting, Natural lighting & dynamic soft shadows
We do compare above technologies and came up with JME which is platform independent. When consider XNA it supports only to windows and Xbox. And when comparing the maturity levels of the XNA and JME, JME shows much efficient and effective performance. Cry engine is one of greatest 3D game engine in terms of detailing. But it need certain level of hardware requirement and it is in the commercial market. So we look at JME which stand alone in both cases.

Project meeting minute - 04

Date & time, duration

The meeting was held on 2010- 10 -21.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members except for H.G.R Dileepa were present there.

Topic of the discussion

1. 3D model mapping from 2D images

2. Human interpretation techniques (with markers/marker less)

3. Virtual environment

Brief Discussion of the discussion

Dr. Chandana Gamage asked us to do the comparison among the available technologies and come up with simple documents which mention about reasonable explanations and comparison etc. Sir asked us to explain research papers read to the group members because that will help to understand the things easily. While doing the research Sir asked us to note down all the novel ideas we have to discuss and figure it out during the meeting sessions. And also asked us to think about possible new theory for add high value to the project. We are supposed to update the wiki page in moodle about the project progress and also Sir asked us to create website for the project and update it as well. While we discuss within group we were asked to do things to broad the discussion such as put ideas and comments on related blogs and discussion forums, etc. Sir mentioned about disabled soldiers in “Ranaviru sevana” where suitable place to do the demonstration of our project. At the end of the discussion we planned about what we do in next week such as continue our previous work carried out on research and prepare the System requirement Specification document.

Key topics discussed

1. 3 design areas defined under topics under discussion

2. extra things to add value to the project

3. plans for the next week

Comments and agreements

1. explain the research papers each one has gone through to the others.

2. create a web site for the project and update it.

3. continue the previous work with reading research papers.

4. prepare the Software Requirements Specification document.

Project "Bubble Boy" Against the Existing Technologies

 The point at which it differs from the rest of the applications is the way in which the complexity of our application is managed. In existing applications most of the things are done using highly sophisticated hardware either embedded into the computer or placed on the user which in turn would cost a lot. In that scenario role of our application comes into play where it will not be required sophisticated hardware on either case. The hidden complexity will be managed by the software and commodity hardware components which will be easily and affordably available will only be there for sake of the purpose. Another notable fact is the kind of the target audience for this system. The target audience is the average set of people who does not have that much computer literacy. So the majority of them find it uncomfortable to continue with that and all what they want is to feel the experience provided by this system with minimal effort without having to struggle severely understand and use the system. Otherwise they are dissatisfied with the system no matter how sophisticated the system is. Since this system could be developed in order to capture such market, this would be great project people who do not have computer literacy.

This innovative research would consider not only the movement, but also the speed of action. Eventually this gives an interaction which is much natural and user friendly. Person throws a spear and a stone to a river are two different activities. But both activities contain almost similar movement. Only change might be the speed.

Implementation

1.1 Software Development Model

During the project, it requires to follow certain software development model in order to achieve better quality output. Since the requirement would affect with the present technology, it is prefer to use Iterative model which is much dynamic. In this project, it can identify a project life cycle (PLC) with five different phases. Those are identification, plan, implementation, testing and complete. As it is planned, new technologies are welcome during any phase. So this eventually becomes highly flexible. Even new ideas could arise while doing implementation. To create more competitive product, it should be up to date. This project basically can be defined as identification of an opportunity in the market. To meet up with these requirements, it is preferred to use an evolutionary/Iterative model as software development model and at the same time team would make sure to adhere with coding paradigms and standards.

1.2 Video Capturing & Analysis

After the research component conduct, it was able to found that there are several possible methods which can be used to achieve this objective but most of them are done using single cameras. In our case we may need to use multiple web cameras in order to capture the 3D motions of the user. Therefore we have to synchronize all those video inputs we capture using those web cameras. Then from that video we take snapshots for analysis user’s gestures. In this case we have to convert 3D in to 2D and then again into 3D, for do this operation we should use effective algorithms for reduce the data loss.

1.3 Image Mapping and Gesture Recognition

Image mapping and gesture recognition, together become an important part of the advanced human computer interaction systems. After capturing the images at a particular rate, as explained in 1.2 its next challenge to analyze them and recognize the 3D gesture of the object that is in front of the camera(s). Here is some research work on this area.

1.3.1 Euclidean Distance Fields

Euclidean Distant field technology helps in estimation of 3D object using a 2D image efficiently. An object’s distance field represents, for any point in space, the distance from that point to the object. The distance can be signed to distinguish between the inside and outside of an object. Estimating Euclidean distance from 2D range images is measured with Projected Distances from 2D image to the camera. Then it provides a 2D grid of line-of-sight distances from the camera to the object.

1.3.2 2D and 3D Analysis for Automated Gesture Recognition

The paper was published for the objective of creating a system that will be helping deaf people communication. It takes three data sets from cameras and analyzes them. The paper explains the technology as set by set. The first data set consists of images acquired with a single camera in a controlled light environment and a segmentation (called “2D segmentation”) based on luminance information. The second data set is acquired with a 3D camera which can produce a depth map; a segmentation (called “3D segmentation”) of the hand configurations based on the video and the depth map is performed. The third data set consists in 3D-segmented masks where the resulting hand mask is warped to compensate for hand pose variations.

1.4 Provide Scenery

In this project we are hoping to deliver a system which would enable interaction without any specific hardware development, but only with aid of software. We need to provide appropriate scene as required by the user. The user will enable to select one of following activities.
• Swimming in an aquarium/sea/river.
• Walking alone by seaside/riverside.
• Walking at a museum
• Drive a racing car
• Walking on moon/space
• Having a walk in a volcanic site
• Tsunami
• Surfing
• Snow board
We can add any kind of scenery since we are hoping to capture the whole body movement. We are hoping to design the graphical environment similar to natural conditions to some extend

1.5 Synchronization of Model and Scenery

According to the user movements the system will capture them. And then the environment should change according to the movement of the user. To estimate the motion we consider overall body movement. And respect to the motion it uses a neural network related thing and try to identify the motion. It will identify as the movement as related to the natural human movements. Then it would consider the real condition and the nature of the current environment. In the natural case, area of the scene changes only when eyes move. Other than that it should be the same angle of the environment. But objects of that area could affect by the user movement.
For example, consider the incident where the user is sitting alone by river side. Then environment should change but area remains the same. When user move head around the viewing angle/area changes. Now user picks a small piece of stone and throws it to the river. If user moves his hand in a vertical manner then it should sink at once. But if he moves hand in a horizontal plane then that piece of stone should jump about two to four times and sink.

Introduction

At a time where there are rapid advances in the field of Human Computer Interaction we have decided to give it another interpretation through the fulfillment of our project. This is one of the major emergent fields in the world of computer technology and people are all the way investigating to come up with advanced interactive interfaces. As it is mentioned above; the project “BubbleBoy" is an innovative, interactive 3D interface using full body motion. Obviously the interaction between the human and computers is done at the user interface which includes both software and hardware.

Sometimes one may think is this something to be invented as there are some applications already of same nature and in that case is there a room for a research component. Yes we also agree with the fact that there are applications with similar behavior as that of ours. But at the same time it should be noted that what we are expecting to do is not same as existing applications.

Nowadays we see lots of advanced user interactive interfaces are coming up to the market. Actually we can find that they are really eye catching and enriched with lot of features once we engage with them. But there are considerable amount of minus points when it is come to communication between computer and human, complexity of hardware and at the same time the software with prevalent systems. The concept of usability of the product is admired most and our main attention in developing the interfaces will be primarily based on that concept. A nice interface does not always mean a user friendly design. It is required badly an interface to be attractive. But at the same time it should fit with the ultimate users expectations. Otherwise the whole purpose of the introducing this product to the customers is lost if the users are not satisfied with the procedures they have to follow in order to facilitate out of this system and they anymore do not want the purchase the product.

That is the pathetic fate of the not user friendly yet unfortunately highly sophisticated products. The best example would be the some current implementations of ATM machines. At the time when the transaction is done the user is frustrated with the whole lot of questionnaire which has come across through the machine. And it is really a bad design of interfaces and a major drawback with most of that system. Likewise there are lots of systems in which they lack the capability of getting things much easier even for an average user and it should be simple enough to a novel person to be adapted with few steps. With contrast to a bad designs if we look into the reason behind the large popularity earned by the mobile phone it is designed in such a manner once the user dial the number of the person whom he or she wants to contact making a call is just a matter of pressing one button without undertaking so much steps.

By taken those facts into consideration in the scope of our project we will make sure that we design the interfaces always pertaining to the fact of ease of use. The interactive techniques we are going to exploit will not consume lot of steps to get to the business. And most importantly system will be configured in a way that has default settings for every stage that user may come across. After exploiting all these features it will be still capable of giving the same experience or even better than that when compared to the applications which use sophisticated hardware devices. For an instance think of a virtual environment which simulate an inside view of a swimming pool. The virtual environment is being subjected to change not only by the way user moves his hand but also by the speed at which user moves his hand. So in order to meet these expectations sound research components will be needed to choose the most optimal technology from rest of the optimal solutions that are available there and we have to come up with our own algorithms undoubtedly and it will realize the objectives of the course module as it is expected.

Moving onto the application areas we can see this concept is exploited as an effective remedial measure in various kinds of fields. Those includes environments for old and disabled people to have experiences which are difficult to have in real, movie industry for animated and real-life motion pictures, the video game industry to create assets for video games the science sector to create highly detailed models of chemical compounds, e-learning systems like virtual tutors to present learning materials, e-commerce systems to have electronic institutions to grab customers, e-tourism to foster tourism where tourists are impersonated as an avatar as well as host of uses depending on the application domain.

The BubbleBoy - voyage through virtual world...

This is a final year course module (CS 4200) project idea planned to do under Computer science and Engineering department of University of Moratuwa

This is a such an innovative framework for applications to give user to real time virtual experience. Using Multi-Move user interface and Using Web cameras detect whole body motions of Player and move the surrounding environment according to those actions. Also we suppose to use cheap components like Web cameras and not to use any tools or devices by user.

Team Name: Yathra

Group Members:

Rajini Dileepa

Hasitha Dilum Gunarathna

Thusitha Kumara
Lahiru Sandaruwan 

.

Project meeting minute - 03

Date & time, duration

The meeting was held on 2010- 10 -14 for about 30 minutes.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

design decisions of several development aspects of the project and Requirements of the project

Brief Discussion of the discussion

In all our sessions with Chandana sir he emphasized the fact that the project we are doing is not just another programming project. Actually it is like a great chance we have given to try out the theoretical aspects of practical scenarios where Engineering concepts are governed. Since this a project carried out by final year under graduate students the standard of the project should be maintained at a certain level. In setting that standard certain aspects like Software Engineering, Engineering design and Engineering Economics have to take into consideration. And at the end of the day we should develop ourselves to a position where we can make design decisions by our own in terms of Engineering. As for the things need to carry out during next week Chandana sir ask us to come up with at least with five approaches to accomplish the task of 2D mapping to 3D. What he said was in selecting those approaches also they should be the best solutions which have been widely used, advanced and efficient. Along with those implementations we can just do test implementations and check whether they behave the way as it is expected. And other research area is to how the environment will react on the changes of the user. And another vital point made by the Chandana sir is to think about the end application at this stage of the development process itself. Although how much effort we have put in our project if the superficial appearance is not that appealing that would not attract the audience and our effort we have put in will not be not appreciated. So in a null shell the visual aspect is something which we can’t avoid and it should be treated with much care.  

Key topics discussed

1. Requirements of the project

2. things to carry out in next week

3. the state of end application

Comments and agreements

This week we agreed to use marker based system due the additional processing needed and the number of cameras that will be needed. What we decided was it should be a maximum of three preferably two.Finally Chandana sir assign each one with some research work for the week.

Project meeting minute - 02

Date & time, duration

The meeting was held on 2010- 10 -07.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Feedback for the draft document

Brief Discussion of the discussion

This contains the summery of the two meetings we have this during this week.Earlier we had meeting on 05th of this month. As for the day we were supposed to send a draft document of a project proposal. And at the time went there Dr. Chandana Gamage reviewed those documents and gave a detailed feedback. If I am to list down the comments made by him Improve on the ability of documentation related tasks · The theme of the project should change from “Advanced” to · “Innovative” What is expected from introduction section of the project proposal. · The points he mentioned were What are the dominant features of our system with compared to other § existing systems What are the qualities of bad designs and instances of them § The target audience of our project is not a set of people who are § highly technical Usability is always admired § We must stick into the fact that how we are going to provide § life-like experience using a computer to the target audience? We have to discuss what are the current implementations with same § behavior and what are the failures and limitations of them The term full body motion is really matters as it conveys the how the § environment is perceived by the user according to his actions and gestures. And another important comment he made was we have to follow some software process model so that it follows a systematic way of developing software product. And according to that we have to have timeline schedule and development phase in turn it would be useful to us also to get to know the health of the project and the current position where we are in it.

Key topics discussed

• Feedback for the draft Project proposal

• References of research papers

• technologies to be used for future developments

• Software process model

Comments and agreements

• Finalize on some software process mode

• not to include too much of technical things to the project proposal at this stage of the project.

• If we select some technology the rationale behind that implementation should be explained clearly

• research component should be carried out in order to find currently existing set of best schemes and what is the best solution out of them considering on pros and cons of each one of them

• put the year of the publication and give a brief introduction as to what it contains in a nut shell when mentioning about a reference

• to keep hard copies of important research papers with us.

• Add revision history to the document

• Improve our skill on writing

• Work on project proposal document according to feedback given

Project meeting minute - 01

Date & time, duration

The first meeting was held on 2010- 10 -01 for about 30 minutes.

Venue

Office room of Dr. Chandana Gamage

Members present

All the members were present there.

Topic of the discussion

Regarding writing the proposal

Brief Discussion of the discussion

Today we basically discuss the topic of what should be the content that should be included in the project proposal. The basic things were listed below.

 • Introduction

 • People involved

 • Theme detail

• This is advanced interactive interface to using full body motion

• Specifying of our system in which it will not have sophisticated hardware but the software

• The complexity will be mostly handled by the software

• Using commodity hardware

• Background research part

• Propose methodology to follow

• Deliverable

• Prototype system with prototype application

• Project schedule

Key topics discussed

1. 4D interactivity (what and how advanced interactive systems are built?)

2. how to combine interface with application framework

3. Application framework (provide scenery and provide motion to scenery changes mapping )

Comments and agreements

Chandana sir asked us to come up with draft project proposal in the next meeting and do the researches on key topic areas he has already mentioned.