A Look at Feet: Recognizing Tailgating via Capacitive Sensing
Distributed, Ambient, and Pervasive Interactions: Technologies and Contexts
International Conference on Distributed, Ambient and Pervasive Interactions (DAPI) <6, 2018, Las Vegas, NV, USA>
At many every day places, the ability to be reliably able to determine how many individuals are within an automated access control area, is of great importance. Especially in high-security areas such as banks and at country borders, access systems like mantraps or drop-arm turnstiles serve this purpose. These automated systems are designed to ensure that only one person can pass through a particular transit area at a time. State of the art systems use camera systems mounted in the ceiling to detect people sneaking in behind authorized individuals to pass through the transit space (tailgating attacks). Our novel method is inspired by recently achieved results in capacitive in-door-localization. Instead of estimating the position of humans, the pervasive capacitance of feet in the transit space is measured to detect tailgating attacks. We explore suitable sensing techniques and sensor-grid layout to be used for that application. In contrast to existing work, we use machine learning techniques for classification of the sensor’s feature vector. The performance is evaluated on hardware-level, by defining its physical effectiveness. Tests with simulated attacks show its performance in comparison with competitive camera-image methods. Our method provides verification of tailgating attacks with an equal-error-rate of 3.5%, which outperforms other methods. We conclude with an evaluation of the amount of data needed for classification and highlight the usefulness of this method when combined with other imaging techniques.
CapMat for Sport Exercise Recognition and Tracking
Darmstadt, TU, Master Thesis, 2018
A large variety of physical exercises can be performed on the ground, typically using a dedicated mat. Many of these exercises do not require additional equipment and mainly consist of specific movements of different body parts. Monitoring sport exercises, i.e. recognition, tracking and counting, has been well researched to help motivate regular exercise and aid in physical rehabilitation. Most of the suggested systems rely on wearable devices or smartphones, which are not always at hand and depend on the limb they are attached to. Camera based solutions are usually not portable and raise privacy concerns.Using dedicated pressure mats has shown great success, but is limited in their adaption to online applications due to their costly computation caused by high sensor resolution. While a few prototypes have been suggested, there is no commercially available product yet, suggesting the difficulty of this area and the need for further research. Furthermore, it is restricted to exercises which are distinguishable by changing the contact area or weight distribution on the mat.We introduce CapMat, a smart sports mat that reaches a user independent recognition rate of 93.5 % in a user study with 9 subjects performing 8 exercises. It is developed with the Open Cap Sense (OCS) board with 8 copper plates as electrodes for capacitive proximity sensing hidden beneath a common sports mat. Moreover, we demonstrate its ability to count exercise repetitions, achieving 93.8 % repetition recognition rate for 12 exercise sets from a single user. The thesis focuses on robust activity classification and several methods to reach this objective are discussed, such as model selection, feature selection and data augmentation.The goal is to develop a system using capacitive sensing to recognize a wide range of exercises. The process from choosing electrode materials to their placement beneath the mat is discussed. The system allows the usage in real-time applications, which is demonstrated with a simple web application running on a Raspberry Pi 3.
Fitness Activity Recognition on Smartphones Using Doppler Measurements
Quantified Self has seen an increased interest in recent years, with devices including smartwatches, smartphones, or other wearables that allow you to monitor your fitness level. This is often combined with mobile apps that use gamification aspects to motivate the user to perform fitness activities, or increase the amount of sports exercise. Thus far, most applications rely on accelerometers or gyroscopes that are integrated into the devices. They have to be worn on the body to track activities. In this work, we investigated the use of a speaker and a microphone that are integrated into a smartphone to track exercises performed close to it. We combined active sonar and Doppler signal analysis in the ultrasound spectrum that is not perceivable by humans. We wanted to measure the body weight exercises bicycles, toe touches, and squats, as these consist of challenging radial movements towards the measuring device. We have tested several classification methods, ranging from support vector machines to convolutional neural networks. We achieved an accuracy of 88% for bicycles, 97% for toe-touches and 91% for squats on our test set.
Performing Indoor Localization with Electric Potential Sensing
Journal of Ambient Intelligence and Humanized Computing
Location-based services or smart home applications all depend on an accurate indoor positioning system. Basically one divides these systems into token-based and token-free localization systems. In this work, we focus on the token-free system based on smart floor technology. Smart floors can typically be built using pressure sensors or capacitive sensors. However, these set-ups are often hard to deploy as mechanical or electrical features are required below the surface and even harder to replace when detected a sensor malfunctioning. Therefore we present a novel indoor positioning system using an uncommon form of passive electric field sensing (EPS), which detects the electric potential variation caused by body movement. The EPS-based smart floor set-up is easy to install by deploying a grid of passive electrode wires underneath any non-conductive surfaces. Easy maintenance is also ensured by the fact that the sensors are not placed underneath the surface, but on the side. Due to the passive measuring nature, low power consumption is achieved as opposed to active capacitive measurement. Since we do not collect image data as in visual-based systems and all sensor data is processed locally, we preserve the user’s privacy. The proposed architecture achieves a high position accuracy and an excellent spatial resolution. Based on our evaluation conducted in our living lab, we measure a mean positioning error of only 12.7 cm.
Step by Step: Early Detection of Diseases Using an Intelligent Floor
European Conference on Ambient Intelligence (AmI) <14, 2018, Larnaca, Cyprus>
Lecture Notes in Computer Science (LNCS), 11249
The development of sensor technologies in smart homes helps to increase user comfort or to create safety through the recognition of emergency situations. For example, lighting in the home can be controlled or an emergency call can be triggered if sensors hidden in the floor detect a fall of a person. It makes sense to also use these technologies regarding prevention and early detection of diseases. By detecting deviations and behavioral changes through long-term monitoring of daily life activities it is possible to identify physical or cognitive diseases. In this work, we first examine in detail the existing possibilities to recognize the activities of daily life and the capability of such a system to conclude from the given data on illnesses. Then we propose a model for the use of floor-based sensor technology to help diagnose diseases and behavioral changes by analyzing the time spent in bed as well as the walking speed of users. Finally, we show that the system can be used in a real environment.
Surface Acoustic Arrays to Analyze Human Activities in Smart Environments
European Conference on Ambient Intelligence (AmI) <14, 2018, Larnaca, Cyprus>
Smart Environments should be able to understand a user’s need without explicit interaction. In order to do that, one step is to build a system that is able to recognize and track some common activities of the user. This way, we can provide a system that provides various services for controlling installed appliances and offering help for every day activities. Applying these services in the users’ environment should make his life more comfortable, easier, and safer. In this paper, we will introduce an embedded sensor system using surface acoustic arrays to analyze human activities in a smart environment. We divided basic activity groups ranging from walking, cupboard closing to falling, including their extended sub-activity groups. We expanded walking into walking barefoot, with shoes and with high heels and further extended closing cupboard with three cupboards locating on different positions. We further investigated the usage of single pickup or a combination of 4 pickups with their effect on the recognition precision. We achieved an overall precision of 97.23% with 10-fold cross validation using support vector machine (SVM) for all sub-activity group combined. Even using one pickup only, we can achieve an overall precision of more than 93%, but we can further increase the precision by using a combination of pickups up to 97.23%.
Activity Recognition On Unmodified Consumer Smartphones Via Active Ultrasonic Sensing
Darmstadt, TU, Master Thesis, 2017
Sensor miniaturisation and streaming classification techniques can be used to recognize human behaviours and contexts. This is extremely valuable to realize smart environments, e.g. to support healthy and independent living. The most important parameters to sense include indoor location, gestures, or emergencies like falls. Up to now, activity recognition systems face a number of sensitive drawbacks. For example, camera-based systems induce privacy issues and are costly to deploy. Body-worn systems are inconvenient to wear over long periods of time. Highly visible systems may introduce social stigma and modify the well-known living environment. In this project, we explore the possibility for the use of a new, unobtrusive, physical principle to sense and recognize human activities using off-the-shelf smart-phone. A person's smart-phone is a cornucopia of information. The huge variety of sensors in today's mobile phones makes these devices a prime target for human activity recognition. Our novel approach is to develop a novel activity recognizing system using an unmodified smart-phone. We profit from integrated microphones and loudspeakers without additional hardware components needed. The advantage of this system is therefore that it can be easily installed on a smart-phone and put into action. An android application has already been developed which is able to send a high frequency sound in the near ultrasound range, e.g. 20 kHz. Using the received echo from the microphone, the information caused by movement in midair around the device will be extracted. In this thesis we intend to improve the performance of the existing system with respect to noise cancellation and other classification schemes. In this thesis, we present an android application called Trainer for complex activity recognition. It is built on ultrasense , a mobile application that capitalizes the characters of ultrasound to inspect the surrounding environment. The application is able to send a high frequency signal in the near ultrasound range, e.g. 20 kHz. Using the received echo from the microphone, the information caused by movement in midair around the device will be extracted. Complex activities tagged under home exercises are evaluated using micro-Doppler signatures [mD-signatures]. We propose an algorithm to classify a set of exercises carried out by the user and show that using the Support vector machine classifier we are able to obtain an accuracy of 85% using Principal component analysis and a signature feature introduced in this thesis as a feature.
An Exploratory Study on Electric Field Sensing
European Conference on Ambient Intelligence (AmI) <13, 2017, Malaga, Spain>
Electric fields are influenced by the human body and other conducting materials. Capacitive measurement techniques are used in touch-screens, in the automobile industry, and for presence and activity recognition in Ubiquitous Computing. However, a drawback of the capacitive technology is the energy consumption, which is an important aspect for mobile devices. In this paper we explore possible applications of electric field sensing, a purely passive capacitive measurement technique, which can be implemented with an extremely low power consumption. To cover a wide range of applications, we examine five possible use cases in more detail. The results show that the application is feasible both in interior spaces and outdoors. Moreover, due to the low energy consumption, mobile usage is also possible.
Best Practices to Visualize Activity Data in Mobile Apps
Darmstadt, TU, Master Thesis, 2017
Physical activity and exercise are essential factors to live a healthy life. Fitness trackers have great potential to assist individuals in making healthy changes to their lifestyle. A variety of fitness trackers are available in the market such as fitness apps based on mobile platform, wearable sensors (e.g. smartwatch, armband, wristband), balancing boards (e.g. Wii fit) etc. In this thesis, the focus is on fitness apps based on mobile platform. These apps provide different information and features to the user such as a summary of the physical activity performed, feedback of the activity (e.g. through virtual trainer), exercise plans according to the user's workout routine, user's achievements and many more. Also, fitness apps aim to present a lot of statistical data to the users regarding their current or previous physical activity which may range from days to years. To visualize this data, visual designs such as maps, graphs, images are used. However, very little is known about such visualization schemes and design strategies for fitness data w.r.t engaging users. Furthermore, it is important to know if the provided features in the app are useful. The main objective of this study is to evaluate different visualization schemes used in visualizing fitness data and to explore usability requirements, motivating factors for using mobile fitness apps. For this purpose, a profound research is done in three phases. The first phase focuses on finding expectations of a user from fitness app through a short primary survey in University Gym, the second phase includes designing an extensive user survey and fitness app mock-ups based on the survey findings in the first phase. In the third phase, the designed mock-ups are evaluated by means of the user survey designed in second phase and the survey results are analyzed using statistical test. The study reveals that users find some visualization schemes very useful whereas they do not prefer some visualization schemes at all. Same is the case observed for motivational features e.g. ranking, rewards and other functionalities of the app e.g. workout summary, nutrition information. This thesis concludes with best practices for designing visualization schemes and analysis of user requirements for mobile fitness applications such as integrated feedback, home screen design of the app and some features like data sharing, data export etc. These findings show the way to develop highly usable fitness applications with user-centric design.
Exercise Monitoring On Consumer Smart Phones Using Ultrasonic Sensing
International Workshop on Sensor-based Activity Recognition (iWOAR) <4, 2017, Rostock, Germany>
Quantified self has been a trend over the last several years. An increasing number of people use devices, such as smartwatches or smartphones to log activities of daily life, including step count or vital information. However, most of these devices have to be worn by the user during the activities, as they rely on integrated motion sensors. Our goal is to create a technology that enables similar precision with remote sensing, based on common sensors installed in every smartphone, in order to enable ubiquitous application. We have created a system that uses the Doppler effect in ultrasound frequencies to detect motion around the smartphone. We propose a novel use case to track exercises, based on several feature extraction methods and machine learning classification. We conducted a study with 14 users, achieving an accuracy between 73% and 92% for the different exercises.
Fiber Defect Detection of Inhomogeneous Voluminous Textiles
Mexican Conference on Pattern Recognition (MCPR) <9, 2017, Huatulco, Mexico>
Quality assurance of dry cleaned industrial textiles is still a mostly manually operated task. In this paper, we present how computer vision and machine learning can be used for the purpose of automating defect detection in this application. Most existing systems require textiles to be spread flat, in order to detect defects. In contrast, we present a novel classification method that can be used when textiles are in inhomogeneous, voluminous shape. Normalization and classification methods are combined in a decision-tree model, in order to detect different kinds of textile defects. We evaluate the performance of our system in realworld settings with images of piles of textiles, taken using stereo vision. Our results show, that our novel classification method using key point pre-selection and convolutional neural networks outperform competitive methods in classification accuracy.
Indoor Localization Based on Passive Electric Field Sensing
European Conference on Ambient Intelligence (AmI) <13, 2017, Malaga, Spain>
The ability to perform accurate indoor positioning opens a wide range of opportunities, including smart home applications and location-based services. Smart floors are a well-established technology to enable marker-free indoor localization within an instrumented environment. Typically, they are based on pressure sensors or varieties of capacitive sensing. These systems, however, are often hard to deploy as mechanical or electrical features are required below the surface. They might also have a limited range or not be compatible with different floor materials. In this paper, we present a novel indoor positioning system using an uncommon form of passive electric field sensing, which detects the change in body electric potential during movement. It is easy to install by deploying a grid of passive wires underneath any non-conductive floor surface. The proposed architecture achieves a high position accuracy and an excellent spatial resolution. In our evaluation, we measure a mean positioning error of only 12.7 cm. The proposed system also combines the advantages of very low power consumption, easy installation, easy maintenance, and the preservation of privacy.
New Approaches for Localization and Activity Sensing in Smart Environments
Ambient Assisted Living
Ambient Assisted Living (AAL) <9, 2016, Frankfurt, Germany>
Smart environments need to be able to fulfill the wishes of its occupants unobtrusively. To achieve this goal, it has to be guaranteed that the current state environment is perceived at all times. One of the most important aspects is to find the current position of the in- habitants and to perceive how they move in this environment. Numerous technologies enable such supervision. Particularly challenging are marker-free systems that are also privacy-preserving. In this paper, we present two such systems for localizing inhabitants in a Smart Environment using - electrical potential sensing and ultrasonic Doppler sensing. We present methods that infer location and track the user, based on the acquired sensor data. Finally, we discuss the advantages and challenges of these sensing technologies and provide an overview of future research directions.
Attack Detection in an Autonomous Entrance System using Optical Flow
7th International Conference on Imaging for Crime Detection and Prevention
International Conference on Imaging for Crime Detection and Prevention (ICDP) <7, 2016, Madrid, Spain>
Unstaffed access control portals are becoming more common in high security areas. Existing systems require expensive hardware, or are sensitive to changing environmental conditions. We present a single camera system for a mantrap which is able to verify that only one individual is in the designated transit area. Our novel approach combines optical flow and machine-learning classification. A database was created that consists of images of attempted attacks and regular verification. The results show that our approach provides competitive results and outperforms detection rates in several attack scenarios.
Platypus - Indoor Localization and Identification through Sensing Electric Potential Changes in Human Bodies
Mobile Systems, Applications, and Services
International Conference on Mobile Systems, Applications, and Services (MobiSys) <14, 2016, Singapore>
Platypus is the first system to localize and identify people by remotely and passively sensing changes in their body electric potential which occur naturally during walking. While it uses three or more electric potential sensors with a maximum range of 2 m, as a tag-free system it does not require the user to carry any special hardware. We describe the physical principles behind body electric potential changes, and a predictive mathematical model of how this affects a passive electric field sensor. By inverting this model and combining data from sensors, we infer a method for localizing people and experimentally demonstrate a median localization error of 0.16m. We also use the model to remotely infer the change in body electric potential with a mean error of 8.8% compared to direct contact-based measurements. We show how the reconstructed body electric potential differs from person to person and thereby how to perform identification. Based on short walking sequences of 5 s, we identify four users with an accuracy of 94 %, and 30 users with an accuracy of 75%. We demonstrate that identification features are valid over multiple days, though change with footwear.
A Gesture Recognition Method for Proximity-Sensing Surfaces in Smart Environments
Distributed, Ambient, and Pervasive Interactions
International Conference on Distributed, Ambient and Pervasive Interactions (DAPI) <3, 2015, Los Angeles, CA, USA>
In order to ease the daily activities in life, a growing number of sophisticated embedded systems is integrated into an users environment. People are in need to communicate with the machines embedded in the surroundings via interfaces which should be as natural as possible. A very natural way of interaction can be implemented via gestures. Gestures should be intuitive, easy to interpret and to learn. In this paper, we propose a method for in-the-air gesture recognition within smart environments. The algorithm used to determine the performed gesture is based on dynamic time warping. We apply 12 capacitive proximity sensors as sensing area to collect gestures. The hand positions within a gesture are converted into features which will be matched with dynamic time warping. The gesture carried out above the sensing area are interpreted in realtime. Gestures supported can be used to control various applications like entertainment systems or other home automation systems.
Indoor Localization Based on Electric Potential Sensing
Darmstadt, TU, Master Thesis, 2015
Indoor localization is needed in applications ranging from health care to entertainment. Although approaches based on video cameras have the upper hand in terms of accuracy and maturity, they raise privacy concerns and require heavy computation. Passive electric field sensing represents a low-cost, low-power, non-intrusive alternative for localization, which is investigated in this thesis. A human being naturally generates an electric field when walking. This field carries an ambiguous and nonlinear information about the person's position. The present thesis proposes to combine measurements from several electric field sensors, thus resolving the ambiguity and obtaining a problem similar to trilateration. The method is presented as a detailed analytical model and implemented in a scalable system, the Platypus. The Platypus operates with a commercial sensor, the PS25451 EPIC (electric potential integrated circuit) manufactured by Plessey Semiconductors, which costs less than 10 Euro and consumes about 6mW. In this work, six sensors are fixed on the ceiling of a room, covering an area of 5m2, and the localization method is evaluated with 30 subjects. Results show that individuals walking at a comfortable speed are localized approximately twice each time they make a step, with an average error of 19.1 cm. The thesis contributes an original localization method that can be used in fusion with other systems, such as infrared sensors, to combine their respective strengths. The described analytical models have a large scope and can be adapted to other applications in human movement sensing.
Opportunities and Applications of Ultrasound Sensing on Unmodified Consumer-grade Smartphones
Darmstadt, TU, Master Thesis, 2015
A person's smartphone is a cornucopia of information. Be it personal data extracted from contacts and calendar entries or the current location via GPS. The huge variety of sensors in today's mobile phones makes these devices a prime target for human activity recognition. The smartphone is no longer solely seen as actuator in smart environments, enabling the user to control auxiliary devices and sensors, but can now play a vital part in the network of sensing information itself. Especially in the area of human activity recognition, camera-based or body-worn systems are predominant. While they achieve high accuracy, these methods often suffer from privacy issues or obtrusiveness and consequently social stigma. In this thesis, I present an unobtrusive approach to perceive the vicinity surrounding the phone by leveraging the properties of ultrasound sensing. The device emits ultrasonic waves via its speaker and records the echo via the microphone. By analyzing the received signal, I can deduct certain movements, e.g. gestures performed above the phone, but also more complex motions involving the whole body of the user. I outline various experiments to estimate the feasibility of ultrasound sensing in different scenarios as well as propose an algorithm and mobile application that can classify given gestures and activities performed by the user. The system is able to recognize predefined gestures with an overall accuracy of 81% over six different users and can detect human activities up to 2m away.
Opportunities for Activity Recognition using Ultrasound Doppler Sensing on Unmodified Mobile Phones
iWOAR 2015 - 2nd international Workshop on Sensor-based Activity Recognition and Interaction
International Workshop on Sensor-based Activity Recognition (iWOAR) <2, 2015, Rostock, Germany>
Nowadays activity recognition on smartphones is ubiquitously applied, for example to monitor personal health. The smartphone's sensors act as a foundation to provide information on movements, the user's location or direction. Incorporating ultrasound sensing using the smartphone's native speaker and microphone provides additional means for perceiving the environment and humans. In this paper, we outline possible usage scenarios for this new and promising sensing modality. Based on a custom implementation, we provide results on various experiments to assess the opportunities for activity recognition systems. We discuss various limitations and possibilities when wearing the smartphone on the human body. In stationary deployments, e.g. while placed on a night desk, our implementation is able to detect movements in proximities up to 2m as well as discern several gestures performed above the phone.