Activity Recognition On Unmodified Consumer Smartphones Via Active Ultrasonic Sensing
Darmstadt, TU, Master Thesis, 2017
Sensor miniaturisation and streaming classification techniques can be used to recognize human behaviours and contexts. This is extremely valuable to realize smart environments, e.g. to support healthy and independent living. The most important parameters to sense include indoor location, gestures, or emergencies like falls. Up to now, activity recognition systems face a number of sensitive drawbacks. For example, camera-based systems induce privacy issues and are costly to deploy. Body-worn systems are inconvenient to wear over long periods of time. Highly visible systems may introduce social stigma and modify the well-known living environment. In this project, we explore the possibility for the use of a new, unobtrusive, physical principle to sense and recognize human activities using off-the-shelf smart-phone. A person's smart-phone is a cornucopia of information. The huge variety of sensors in today's mobile phones makes these devices a prime target for human activity recognition. Our novel approach is to develop a novel activity recognizing system using an unmodified smart-phone. We profit from integrated microphones and loudspeakers without additional hardware components needed. The advantage of this system is therefore that it can be easily installed on a smart-phone and put into action. An android application has already been developed which is able to send a high frequency sound in the near ultrasound range, e.g. 20 kHz. Using the received echo from the microphone, the information caused by movement in midair around the device will be extracted. In this thesis we intend to improve the performance of the existing system with respect to noise cancellation and other classification schemes. In this thesis, we present an android application called Trainer for complex activity recognition. It is built on ultrasense , a mobile application that capitalizes the characters of ultrasound to inspect the surrounding environment. The application is able to send a high frequency signal in the near ultrasound range, e.g. 20 kHz. Using the received echo from the microphone, the information caused by movement in midair around the device will be extracted. Complex activities tagged under home exercises are evaluated using micro-Doppler signatures [mD-signatures]. We propose an algorithm to classify a set of exercises carried out by the user and show that using the Support vector machine classifier we are able to obtain an accuracy of 85% using Principal component analysis and a signature feature introduced in this thesis as a feature.
An Exploratory Study on Electric Field Sensing
Braun, Andreas (Ed.) et al.: Ambient Intelligence : 13th European Conference, AmI 2017. Springer, 2017. (Lecture Notes in Computer Science (LNCS) 10217), pp. 247-262
European Conference on Ambient Intelligence (AmI) <13, 2017, Malaga, Spain>
Electric fields are influenced by the human body and other conducting materials. Capacitive measurement techniques are used in touch-screens, in the automobile industry, and for presence and activity recognition in Ubiquitous Computing. However, a drawback of the capacitive technology is the energy consumption, which is an important aspect for mobile devices. In this paper we explore possible applications of electric field sensing, a purely passive capacitive measurement technique, which can be implemented with an extremely low power consumption. To cover a wide range of applications, we examine five possible use cases in more detail. The results show that the application is feasible both in interior spaces and outdoors. Moreover, due to the low energy consumption, mobile usage is also possible.
Best Practices to Visualize Activity Data in Mobile Apps
Darmstadt, TU, Master Thesis, 2017
Physical activity and exercise are essential factors to live a healthy life. Fitness trackers have great potential to assist individuals in making healthy changes to their lifestyle. A variety of fitness trackers are available in the market such as fitness apps based on mobile platform, wearable sensors (e.g. smartwatch, armband, wristband), balancing boards (e.g. Wii fit) etc. In this thesis, the focus is on fitness apps based on mobile platform. These apps provide different information and features to the user such as a summary of the physical activity performed, feedback of the activity (e.g. through virtual trainer), exercise plans according to the user's workout routine, user's achievements and many more. Also, fitness apps aim to present a lot of statistical data to the users regarding their current or previous physical activity which may range from days to years. To visualize this data, visual designs such as maps, graphs, images are used. However, very little is known about such visualization schemes and design strategies for fitness data w.r.t engaging users. Furthermore, it is important to know if the provided features in the app are useful. The main objective of this study is to evaluate different visualization schemes used in visualizing fitness data and to explore usability requirements, motivating factors for using mobile fitness apps. For this purpose, a profound research is done in three phases. The first phase focuses on finding expectations of a user from fitness app through a short primary survey in University Gym, the second phase includes designing an extensive user survey and fitness app mock-ups based on the survey findings in the first phase. In the third phase, the designed mock-ups are evaluated by means of the user survey designed in second phase and the survey results are analyzed using statistical test. The study reveals that users find some visualization schemes very useful whereas they do not prefer some visualization schemes at all. Same is the case observed for motivational features e.g. ranking, rewards and other functionalities of the app e.g. workout summary, nutrition information. This thesis concludes with best practices for designing visualization schemes and analysis of user requirements for mobile fitness applications such as integrated feedback, home screen design of the app and some features like data sharing, data export etc. These findings show the way to develop highly usable fitness applications with user-centric design.
Exercise Monitoring On Consumer Smart Phones Using Ultrasonic Sensing
Yordanova, Kristina (Ed.) et al.: iWOAR 2017 : 4th international Workshop on Sensor-based Activity Recognition and Interaction. New York: ACM Press, 2017. (ACM International Conference Proceedings Series 1183), Article No. 9, 6 p.
International Workshop on Sensor-based Activity Recognition (iWOAR) <4, 2017, Rostock, Germany>
Quantified self has been a trend over the last several years. An increasing number of people use devices, such as smartwatches or smartphones to log activities of daily life, including step count or vital information. However, most of these devices have to be worn by the user during the activities, as they rely on integrated motion sensors. Our goal is to create a technology that enables similar precision with remote sensing, based on common sensors installed in every smartphone, in order to enable ubiquitous application. We have created a system that uses the Doppler effect in ultrasound frequencies to detect motion around the smartphone. We propose a novel use case to track exercises, based on several feature extraction methods and machine learning classification. We conducted a study with 14 users, achieving an accuracy between 73% and 92% for the different exercises.
Fiber Defect Detection of Inhomogeneous Voluminous Textiles
Carrasco-Ochoa, Jesús Ariel (Ed.) et al.: Pattern Recognition : 9th Mexican Conference, MCPR 2017. Springer, 2017. (Lecture Notes in Computer Science (LNCS) 10267), pp. 278-287
Mexican Conference on Pattern Recognition (MCPR) <9, 2017, Huatulco, Mexico>
Quality assurance of dry cleaned industrial textiles is still a mostly manually operated task. In this paper, we present how computer vision and machine learning can be used for the purpose of automating defect detection in this application. Most existing systems require textiles to be spread flat, in order to detect defects. In contrast, we present a novel classification method that can be used when textiles are in inhomogeneous, voluminous shape. Normalization and classification methods are combined in a decision-tree model, in order to detect different kinds of textile defects. We evaluate the performance of our system in realworld settings with images of piles of textiles, taken using stereo vision. Our results show, that our novel classification method using key point pre-selection and convolutional neural networks outperform competitive methods in classification accuracy.
Indoor Localization Based on Passive Electric Field Sensing
Braun, Andreas (Ed.) et al.: Ambient Intelligence : 13th European Conference, AmI 2017. Springer, 2017. (Lecture Notes in Computer Science (LNCS) 10217), pp. 64-79
European Conference on Ambient Intelligence (AmI) <13, 2017, Malaga, Spain>
The ability to perform accurate indoor positioning opens a wide range of opportunities, including smart home applications and location-based services. Smart floors are a well-established technology to enable marker-free indoor localization within an instrumented environment. Typically, they are based on pressure sensors or varieties of capacitive sensing. These systems, however, are often hard to deploy as mechanical or electrical features are required below the surface. They might also have a limited range or not be compatible with different floor materials. In this paper, we present a novel indoor positioning system using an uncommon form of passive electric field sensing, which detects the change in body electric potential during movement. It is easy to install by deploying a grid of passive wires underneath any non-conductive floor surface. The proposed architecture achieves a high position accuracy and an excellent spatial resolution. In our evaluation, we measure a mean positioning error of only 12.7 cm. The proposed system also combines the advantages of very low power consumption, easy installation, easy maintenance, and the preservation of privacy.
New Approaches for Localization and Activity Sensing in Smart Environments
Wichert, Reiner (Ed.) et al.: Ambient Assisted Living : 9. AAL-Kongress 2016 Frankfurt/M. Springer International Publishing, 2017. (Advanced Technologies and Societal Change), pp. 73-84
Ambient Assisted Living (AAL) <9, 2016, Frankfurt, Germany>
Smart environments need to be able to fulfill the wishes of its occupants unobtrusively. To achieve this goal, it has to be guaranteed that the current state environment is perceived at all times. One of the most important aspects is to find the current position of the in- habitants and to perceive how they move in this environment. Numerous technologies enable such supervision. Particularly challenging are marker-free systems that are also privacy-preserving. In this paper, we present two such systems for localizing inhabitants in a Smart Environment using - electrical potential sensing and ultrasonic Doppler sensing. We present methods that infer location and track the user, based on the acquired sensor data. Finally, we discuss the advantages and challenges of these sensing technologies and provide an overview of future research directions.
Attack Detection in an Autonomous Entrance System using Optical Flow
7th International Conference on Imaging for Crime Detection and Prevention : ICDP 2016 [online]. [cited 11 July 2017] Available from: [http://digital-library.theiet.org/content/conferences/2016/0006], 2016, 6 p.
International Conference on Imaging for Crime Detection and Prevention (ICDP) <7, 2016, Madrid, Spain>
Unstaffed access control portals are becoming more common in high security areas. Existing systems require expensive hardware, or are sensitive to changing environmental conditions. We present a single camera system for a mantrap which is able to verify that only one individual is in the designated transit area. Our novel approach combines optical flow and machine-learning classification. A database was created that consists of images of attempted attacks and regular verification. The results show that our approach provides competitive results and outperforms detection rates in several attack scenarios.
Platypus - Indoor Localization and Identification through Sensing Electric Potential Changes in Human Bodies
Balan, Rajesh (Ed.) et al.: Mobile Systems, Applications, and Services : Proceedings of the 14th Annual International Conference on. New York: ACM, 2016, pp. 17-30
International Conference on Mobile Systems, Applications, and Services (MobiSys) <14, 2016, Singapore>
Platypus is the first system to localize and identify people by remotely and passively sensing changes in their body electric potential which occur naturally during walking. While it uses three or more electric potential sensors with a maximum range of 2 m, as a tag-free system it does not require the user to carry any special hardware. We describe the physical principles behind body electric potential changes, and a predictive mathematical model of how this affects a passive electric field sensor. By inverting this model and combining data from sensors, we infer a method for localizing people and experimentally demonstrate a median localization error of 0.16m. We also use the model to remotely infer the change in body electric potential with a mean error of 8.8% compared to direct contact-based measurements. We show how the reconstructed body electric potential differs from person to person and thereby how to perform identification. Based on short walking sequences of 5 s, we identify four users with an accuracy of 94 %, and 30 users with an accuracy of 75%. We demonstrate that identification features are valid over multiple days, though change with footwear.
A Gesture Recognition Method for Proximity-Sensing Surfaces in Smart Environments
Streitz, Norbert (Ed.) et al.: Distributed, Ambient, and Pervasive Interactions : DAPI 2015. Springer International Publishing, 2015. (Lecture Notes in Computer Science (LNCS) 9189), pp. 163-173
International Conference on Distributed, Ambient and Pervasive Interactions (DAPI) <3, 2015, Los Angeles, CA, USA>
In order to ease the daily activities in life, a growing number of sophisticated embedded systems is integrated into an users environment. People are in need to communicate with the machines embedded in the surroundings via interfaces which should be as natural as possible. A very natural way of interaction can be implemented via gestures. Gestures should be intuitive, easy to interpret and to learn. In this paper, we propose a method for in-the-air gesture recognition within smart environments. The algorithm used to determine the performed gesture is based on dynamic time warping. We apply 12 capacitive proximity sensors as sensing area to collect gestures. The hand positions within a gesture are converted into features which will be matched with dynamic time warping. The gesture carried out above the sensing area are interpreted in realtime. Gestures supported can be used to control various applications like entertainment systems or other home automation systems.
Indoor Localization Based on Electric Potential Sensing
Darmstadt, TU, Master Thesis, 2015
Indoor localization is needed in applications ranging from health care to entertainment. Although approaches based on video cameras have the upper hand in terms of accuracy and maturity, they raise privacy concerns and require heavy computation. Passive electric field sensing represents a low-cost, low-power, non-intrusive alternative for localization, which is investigated in this thesis. A human being naturally generates an electric field when walking. This field carries an ambiguous and nonlinear information about the person's position. The present thesis proposes to combine measurements from several electric field sensors, thus resolving the ambiguity and obtaining a problem similar to trilateration. The method is presented as a detailed analytical model and implemented in a scalable system, the Platypus. The Platypus operates with a commercial sensor, the PS25451 EPIC (electric potential integrated circuit) manufactured by Plessey Semiconductors, which costs less than 10 Euro and consumes about 6mW. In this work, six sensors are fixed on the ceiling of a room, covering an area of 5m2, and the localization method is evaluated with 30 subjects. Results show that individuals walking at a comfortable speed are localized approximately twice each time they make a step, with an average error of 19.1 cm. The thesis contributes an original localization method that can be used in fusion with other systems, such as infrared sensors, to combine their respective strengths. The described analytical models have a large scope and can be adapted to other applications in human movement sensing.
Opportunities and Applications of Ultrasound Sensing on Unmodified Consumer-grade Smartphones: Möglichkeiten und Anwendungen von Ultraschall Messungen auf nicht modifizierten Smartphones für Endverbraucher
Darmstadt, TU, Master Thesis, 2015
A person's smartphone is a cornucopia of information. Be it personal data extracted from contacts and calendar entries or the current location via GPS. The huge variety of sensors in today's mobile phones makes these devices a prime target for human activity recognition. The smartphone is no longer solely seen as actuator in smart environments, enabling the user to control auxiliary devices and sensors, but can now play a vital part in the network of sensing information itself. Especially in the area of human activity recognition, camera-based or body-worn systems are predominant. While they achieve high accuracy, these methods often suffer from privacy issues or obtrusiveness and consequently social stigma. In this thesis, I present an unobtrusive approach to perceive the vicinity surrounding the phone by leveraging the properties of ultrasound sensing. The device emits ultrasonic waves via its speaker and records the echo via the microphone. By analyzing the received signal, I can deduct certain movements, e.g. gestures performed above the phone, but also more complex motions involving the whole body of the user. I outline various experiments to estimate the feasibility of ultrasound sensing in different scenarios as well as propose an algorithm and mobile application that can classify given gestures and activities performed by the user. The system is able to recognize predefined gestures with an overall accuracy of 81% over six different users and can detect human activities up to 2m away.
Opportunities for Activity Recognition using Ultrasound Doppler Sensing on Unmodified Mobile Phones
Matthies, Denys J.C. (Ed.) et al.: iWOAR 2015 : 2nd international Workshop on Sensor-based Activity Recognition and Interaction. New York: ACM Press, 2015, 10 p.
International Workshop on Sensor-based Activity Recognition (iWOAR) <2, 2015, Rostock, Germany>
Nowadays activity recognition on smartphones is ubiquitously applied, for example to monitor personal health. The smartphone's sensors act as a foundation to provide information on movements, the user's location or direction. Incorporating ultrasound sensing using the smartphone's native speaker and microphone provides additional means for perceiving the environment and humans. In this paper, we outline possible usage scenarios for this new and promising sensing modality. Based on a custom implementation, we provide results on various experiments to assess the opportunities for activity recognition systems. We discuss various limitations and possibilities when wearing the smartphone on the human body. In stationary deployments, e.g. while placed on a night desk, our implementation is able to detect movements in proximities up to 2m as well as discern several gestures performed above the phone.