an elderly care system based on multiple information fusion.
1. Aging population is a global problem. 1] Especially in China. Most of the children who are busy with their work have little time to take care of their parents and have a lot of pressure on their support. As most elderly people become empty nesters, monitoring their living conditions is not only to solve family problems, but also to solve social problems. China\'s senior welfare studies are still in the initial stage of development [2, 3]. Existing products based on wearable sensors can sometimes feel inconvenient and easily forget to carry them. Products based on audio sensors can judge the living conditions of the elderly through sound signals, but are vulnerable to environmental noise, resulting in low accuracy. The products based on visual sensors also have problems such as limited visual acquisition and privacy disclosure. Therefore, the development of a set of elderly care monitoring system that meets the requirements of privacy protection has important significance in family, society, practice and other aspects. The system should be able to monitor daily life effectively and properly assess the health status of the elderly. When an accident occurs, an alarm signal is issued to inform family members or other relevant personnel. Some related systems are proposed in the literature [4-8]. For example, Kidd and others.  This paper proposes a \"perception family\" system, which captures the real Shooting the time image of the elderly through the camera, the children can see the current activity information of the elderly through the Internet, can view the recorded information, and better understand the situation of the elderly. Recently, Khosla and others.  An interactive intermodal social robot system was reported for improving the quality of care for elderly people in nursing homes in Australia. In their system, they use multimodal interaction ( Voice, gestures, emotions, touch screen and dance) Social robots. Suradwala and mukopadi Hai [1  A wireless sensor network is proposed. Health measurement of the elderly based on the home monitoring system. In their system, they use a number of interconnected sensors to detect the use of electrical equipment, the use of beds and chairs, and emergency buttons, wireless sensor networks are made up of different types of sensors, such as electrical and force, and contact sensors with Zigbee module sensing units are installed in older homes. This paper proposes a pension system based on multi-agent. Information fusion technology, with video processing technology as the core, combines sound detection, infrared detection, pulse detection and other technologies. Specifically, the system uses dsp arm dual Core board with OMAP L138 as processor. A six- In order to simplify the circuit and reduce the cost, layer PCB is applied and designed. Through background modeling methods and updates, people who move at the front desk can get the appropriate help for a month- Connection analysis and shadow elimination. Using some features of the smallest external rectangle, drop- Falls in the elderly can be detected appropriately. In order to obtain the life information of the elderly without revealing privacy, several information acquisition boards equipped with infrared sensors, laser sensors, sound sensors and pulse sensors were used at the door, in the toilet, in the bedroom. The multi-information fusion mechanism improves the efficiency and accuracy of the pension system. Therefore, the developed system can accurately track the indoor location of the elderly, detect abnormal activities, and automatically notify relatives when an accident occurs. In general, this study undoubtedly provides an important basis for the promotion and application of the elderly care system. The rest of the paper is organized as follows. The second section describes the whole system in detail. Section 3 video analysis-based falling- The downlink detection algorithm is given. See Section 4 for system setup and experimental results. Finally, conclusions and discussions are given in part 5. 2. System Description 2. 1. System Overview. The schematic diagram of the whole system is shown in Figure 1. The whole system consists of a motherboard and several information collection boards. The hardware system is based on the main control board and the information collection Board. The information collection Board is installed in the correct position around the room. These information collection boards then collect sound, infrared and pulse data directly, as well as some living conditions of the elderly, such as whether he/she is absent or has abnormal sleep, which can then be easily obtained. Through the video analysis of the main board, we can find out whether the elderly fall or not. When all these living conditions are obtained, upload them to the server over Ethernet. Parents can view the real- Through the special app installed on the phone, the time status and historical status of the elderly. On the other hand, when the system detects an unexpected situation, a text message will be automatically sent to relatives through GPRS ( General Packet Radio Service) Modules installed on the system. 2. 2. Design of hardware system. Three aspects should be considered for hardware design. First of all, the hardware system works in the indoor environment, the main impact is the temperature and weather changes. Secondly, the camera is fixed, so the video analysis algorithm has certain robustness to simple noise interference. Third, the system should be in the actual work. time. In view of these aspects, we build a DSP-based ( Digital Signal Processor+ ARM ( Set of Oak)dual-core CPU ( Central processing unit)with OMAP ( Open Multimedia Application Platform)L138  As a processor developed by Texas Instruments and three infrared and sound detection modules ( Information acquisition control panel) A pulse detection module and two analog cameras are used as sensors. The hardware platform has powerful data processing capability, which meets the real-time requirements of the system. Time and efficiency requirements. Figure 2 shows the schematic diagram of the system hardware platform. (1) The core of the motherboard is OMAP L138, which is a doublecore (DSP + ARM) CPU with up to 456 MHz operating frequency, 512 MB extended NAND flash memory, and 128 MB DDR2 ( Double Data Rate 2) Memory, as well as rich external interfaces such as Ethernet, video and LCD ( LCD display) Interface, etc. The main control board adopts the wireless transceiver module yb30_si4432 developed by Silicon Laboratory (silabs. com) Communicate with the information acquisition board. The information acquisition board has the features of long transmission distance, low cost, high integration and strong wall-crossing ability. The living conditions of the elderly when they fall and the images taken are sent through this module. (2) There are 4 pieces of information collection Board, which uses the MCU of the micro-controller  It was developed by ST Micro-electronics. Three of them are equipped with infrared, sound and laser detection modules installed in the toilet, bedroom and gate respectively to detect abnormal conditions including long periods of time Stay in the toilet, don\'t go home, or time for sleep disorders. As mentioned earlier, once an exception occurs, the data will be transmitted to the main control board through the wireless transceiver module. The last piece is used to detect the pulse frequency of the elderly. It can be carried with you or measured when needed. (3) The main control board also judges the abnormal state of the living room when the elderly fall, or detects the abnormal sleep state of the bedroom through another infrared camera. (4)The dual- The core Main Control Board sends the collected information to the server through Ethernet transmission. The server saves the data to the database and displays the current status of the elderly. 2. 3. Design of software system. The software system in this paper consists of four parts: Main Control Board software, information acquisition board software, server software and client software. The functions of the main control board software include abnormal state judgment, notification warning, fall detection, etc. The functions of the information acquisition board software include infrared detection, laser detection, sound detection, pulse detection, etc. The main function of the server software is to do database operations, and the client software is the interface for relatives to view/view the status of the elderly. The schematic diagram of the whole software is shown in figure 3. 2. 3. 1. Design of main control board software. The software of the main control board is divided into three layers from bottom to top: Peripheral Driver function layer, multi-task function layer and core algorithm layer. The driver function layer is used to initialize the periphery of the main control board with CSL ( Chip support Library) The firmware library that drives them to work properly. The multi-task function layer is designed to realize different tasks of the system, including video acquisition, video display, timer interrupt, wireless sending and receiving, SMS sending, server communication and other tasks. The core algorithm layer mainly completes the fall detection of the elderly. The software framework of the main control board is shown in figure 4. According to the software framework and combined with the features of the SYS/BIOS multitasking operating system ( Operating System) The software workflow of the main control board is shown in figure 5. After the motherboard is powered on, the system will first do some initialization work, such as system clock configuration, CSL Library initialization, peripheral and memory resource initialization, etc. The details are as follows :(1) System clock configuration: Set the OMAP L138 system clock to 456 MHz (2) Peripheral initialization: GPIO ( General input/output) Port initialization, driver initialization of lan87 10A, EEPROM initialization, driver initialization of si4432, driver initialization of timer1, driver initialization of tvp5150, at070tn833) Memory resource management: map the appropriate data to DDR2 (4) Read and set the parameters in the EEPROM (5) Multi-task operation: Parameter Setting task, drop behavior detection task ( Including target detection, target tracking, fall behavior recognition, etc) , SMS task, sleep anomaly judgment task, wireless communication, network communication task set \"mode flag\" in figure 5 through hardware DIP switch \". When the mode flag \"ModeSet\" is 1, the system is set to parameter setting mode. In this mode, managers can modify a series of parameters such as room number, the mobile phone number and threshold in the video analysis algorithm, even the IP address obtained through PC software. When \"ModeSet\" is 0, the system is set to work mode. 2. 3. 2. Design of information acquisition board software. As mentioned earlier, we use the information collection board to collect and judge the abnormal life information of the gate, toilet and bedroom. The software diagram of the information acquisition board is shown in figure 6. It has three layers, that is, the peripheral driver function layer, the interface function layer, and the control algorithm layer. The Peripheral Driver function layer is initialized and the peripheral device of the driver 332 is driven. Timer TIMER2 is initialized to a universal timer that generates an interrupt every 1 second to receive and determine the various signals and changes from the sensor. IN/OUT detection module communicates with stm through RS485 bus and interrupts the IN/OUT status of the elderly through serial port. Sound, light and infrared sensors receive data through external interrupts to jointly judge the living conditions of the elderly. The SI4432 wireless module communicates data with the stm connection through the SPI bus interface. The committee will send information on the status of the elderly to the main control committee. The three life information that the elderly at the door should detect includes (01), at home (02) Go out without going home (03); The three life information of the elderly in the bedroom includes getting up (04), sleeping (05) Abnormal sleep (06); Abnormal toilet (07) Is the only state in the toilet that should be detected. Each state is obtained through multi-sensor fusion. Let\'s take the information collected in the bedroom as an example to explain the logic of the information fusion system :(1) The entrance detection sensor detects the elderly entering the bedroom. (2) If the light sensor module detects that the elderly on the bed are moving, and the current time is the sleep time of the elderly, the system judges that the elderly start to sleep. (3) When the elderly are sleeping, but in more than 20 seconds, the sound and infrared sensors cannot detect any valid data, and the system will suspect that there will be abnormal sleep. (4) In sleep mode, the entry detection sensor module detects valid data, if the current time is the wake-up time, then When the time comes, the system will judge the old man getting up. 2. 3. 3. Design of client software. The client software can be divided into two parts, PC ( PC) Monitor client and mobile client. (1) The PC monitoring client displays data from the database server, including the living status of the elderly received from the motherboard and the anomalies that occur over time. The PC-side software was developed with Visual Studio 2013. The development language is C and the database is sqlsever2012. (2) In order to adapt to the two most widely used mobile phone operating systems, namely [Android]11] And iOS, two mobile phone clients were developed respectively. The Android client software development platform is Eclipse 4. 5. Develop the language in JAVA, the iPhone client development platform is Mac OS X, and the language is ObjectiveC. The mobile phone client includes the login interface, the status interface, and the message list interface. The user needs to enter the correct user name and password when logging in, which to some extent protects the privacy of the elderly. The server records all the information into the database; When the user logs in to the mobile client to query the current status or image of the elderly, the server will package it in the standard Json format and send it to the mobile phone, and then the mobile client will parse the received package, and print a list of messages for users to view. If a falling- If the next event occurs, relatives can check- Download pictures through the mobile client. The PC client and the mobile client communicate according to the flow chart shown in Figure 7. 3. Video Analysis-Based Falling- Fall Detection- Down detection can be completed in many ways [12, 13]. In this paper, we use the method based on video analysis. down detection. Video analysis- Object-based detection is widely used in many fields. 14]. Taking into account the privacy of the elderly, the camera in the living room and bedroom is unable to take real photostime videos. Only when the old man falls down can the abnormal image be viewed through the mobile client. In this paper, The next detection is divided into three steps: mobile person- Detection, shadow elimination and descent- Feature extraction. 3. 1. Human body detection. We use background subtraction [15-17] For the detection of sports human body. There are three main steps in Background Subtraction Detection based on moving objects: Background Modeling, background update, and background subtraction. To get a real An improved background modeling method for time motion human body detection is proposed; Background update algorithm combined with Surendra  We can get a quick and accurate test. 3. 1. 1. Improvement of background modeling algorithm. Background Modeling refers to extracting the background from the video sequence, which is the key and basic step in the background subtraction algorithm. An improved background modeling algorithm based on frame difference method is proposed. The core idea is to threshold the difference image to update the initial background ( The initial gray value is 0) Until the background is established. Considering that the movement of the elderly is relatively slow, the relative movement of adjacent frames is very small or even static, which will lead to unsatisfactory results. Therefore, in this paper, the two images that do differential operations are not reflective of the improved adjacent frames. The core of the algorithm is to compare the gray scale of pixels in the same position in two images at different times. When the threshold is less than a certain value, treat it as a background point. The specific steps are as follows: the first step. Take out five frames with one F Frame per two frames ( F = 15) Separation in the original sequence. Initialize the gray value of the background image to 0. Step 2. The difference image is obtained from the first frame and the second frame by image subtraction. If the gray value of pixels in the difference image is less than the obvious threshold T ( T = 20 in this article) At the same time, the gray value of the pixel in the background image is 0, and the gray value of the pixel in the background image is set to the same value as the second frame Hx. [ Non-reproducible mathematical expressions]. (1)Step 3. Repeat step 2 to bring the second and third frame images into the calculation until all 5 images are completed and the background image is established. Step 4. Update the established background image in real time: subtract the current image and the background image obtained in step 3, and then get the differential image. If the value of the pixel in this difference image is greater than the threshold T1 (here is 25) Then update the value to 255. We declare this as moving pixels without updating the background. Otherwise, you need to update the background image to binary image 0. [ Non-reproducible mathematical expressions]. (2)3. 1. 2. Surendra background update algorithm. The background template image is the initial background image of the previous F * 5 frame image, but the background image of the latter frame is not static. Due to the influence of light and other objects, in order to adapt to the changes of indoor light and other environmental factors, the background template needs to be updated in real time. Since the background image is required for this step, only still pixels need to be updated. In this article, we use the weight formula below to update the background, which is the Surendra background update algorithm. BeiJ [B. sub. H](x, y), [B. sub. S](x, y), and [B. sub. V](x, y) H, S and V components of background image B (x, y); and Shadow(x, y) Is a binary image that shows whether the Pixel is a shadow point. The greater the brightness of the environment, the smaller [lambda]is; [delta] It is a parameter set to avoid too many points being mistaken for Shadow points to enhance robustness. [lambda]and [delta]satisfy 0 < [lambda]< [delta]< 1. The shadow is saturated and the color change is not obvious, so 1> [[alpha]. sub. S]> 0. A value [for the sake of more satisfactory test results [[alpha]. sub. H] Added to the limit, can be adjusted according to the specific application scenario. The effect of shadow removal is shown in Figure 10. The video sequence is the same as in figure 8. In Figure 9, the first column is 8- Community connectivity analysis. The second column is the detected shadow, and the third column is the final foreground image after the shadow is removed. We can see that the shadows are removed perfectly. 3. 3. Falling- Feature extraction. A falling- Fall means that the old man suddenly fell down and did not stand up for a period of time. In this article, we use the smallest external rectangle Feature extraction. The minimum external rectangle is the rectangle that contains the smallest area that is a bit in an area. For falling- This area is the foreground human body divided. The red color in Figure 11 shows the sketch of the qualified rectangle. [Four parameters] in Figure 11]X. sub. max], [X. sub. min], [Y. sub. max], and [Y. sub. min] Need to draw a rectangle, the rectangle is x-axis and the y-axis. As can be seen from Figure 11 ,[X. sub. min] Is the x position of the leftmost point in 8- Connect the area, and [X. sub. max] It\'s the right-most point in 8. Connect area. [There are similar meanings]Y. sub. max]and [Y. sub. min]. As shown in Figure 12, the aspect ratio of the minimum external rectangle changes quickly when the elderly fall, so this aspect ratio can be used as a feature of the fall --down detection. However, if the old man is too close or too far from the camera, the aspect ratio will cause the detection to fail. On the other hand, when the elderly fall, the biggest change is x- Axis direction, and y- The axis direction of the center of gravity does not change much. Therefore, in this paper, we will have the aspect ratio K, the absolute slope of the mass center [ Absolute value of S] , And the center of gravity in x-Axis direction]X. sub. mid] To determine if it\'s down- The calculation is as follows :(1) Aspect ratio K: K = [Y. sub. max]-[Y. sub. min]/[X. sub. max]-[X. sub. min]. (9)2) Slope of center of mass S: S = [X. sub. c]-[X. sub. min]/[Y. sub. c]-[Y. sub. min], (10)where [X. sub. c]and [Y. sub. c] Defined in [ Non-reproducible mathematical expressions]. (11)In (11), [n. sub. x] Number of foreground pixels in object xth column ,[n. sub. y] Is the number of foreground pixels in line yth, and n is the total number of foreground pixels. (3) Center of gravity in X-Axis direction]X. sub. mid]: [X. sub. mid]= [[summation]. sub. (x,y)[member of]H][x. sup. *]H(x, y)/[[summation]. sub. (x,y)[member of]H]H(x, y). (12)In (12), H(x, y) The gray value of moving objects in pixels in rectangular frames (x, y). When the old man fell, K 1, and [X. sub. mid] Great changes. Figure 13 shows several examples of the minimum external rectangles obtained according to the method presented in this paper. 4. According to the function of the elderly care system, the whole system is designed and implemented. As mentioned earlier, the system is divided into two parts: the main control board and the information collection Board. The embedded platform designed is shown in Figure 14. In Figure 14, right- Hand-side Main Control Board for video analysis and multi- Information Fusion. The left- The hand side is the information collection Board, which is placed at the door, toilet and bedroom to collect information about the life of the elderly. The hardware platform uses the algorithm described earlier and has been fully tested. The experimental results are shown in Table 1. As can be seen from Table 1, the eight functions of the system are tested. In Table 1, it is easy to get the status \"out\", \"out no back\", \"at home\" and \"toilet abnormality\" using the infrared thermoelectric sensor module \", by detecting the direction of movement of the elderly. Here, \"toilet abnormality\" means that the elderly spend too long in the toilet. The sound detection module installed on the bed can get \"sleep\" and \"sleep abnormality \". \"Abnormal sleep\" state means that the elderly may not be able to breathe for a certain period of time. \"Get Up\" and \"fall \"- As mentioned earlier, the down \"status is obtained through video analysis. The experimental results show that the system meets the design requirements. 5. In the conclusion and discussion of this paper, we propose a system of elderly care based on multi-information fusion. Experiments have proved that when an accident occurs in the elderly, the system can effectively notify relatives through GPRS. Moreover, the system can provide an interface for relatives to inquire about the elderly\'s living conditions by installing the app on the elderly\'s mobile phone. In general, the developed system has the features of fast response, high precision and low cost, and can meet the requirements of real-time. Time monitoring of the living conditions of the elderly can be widely used in home care. In future work, the system can be further improved by integration with other wearable sensors, such as ECG sensors  This can provide more information about the living conditions of the elderly. Conflict of Interest authors declare they have no conflict of interest. The work was partially supported by the National Natural Science Foundation of China. 61401129. Natural Science Fund of Zhejiang province (LY17F010020). References A. Wong, transmission of consumption by population aging and monetary policy, 2015, website/default/File2g9f9ga. pdf. B. Zhejiang University of Technology, Ma, a monitoring system for elderly people living alone based on integrated computer vision, 2014. Y. Bai, J. Li, and J. Design of fall detection system based on embedded video surveillance, TV Technology, Volume 138, no. 15, 2014. L. Liu, E. Stroulia, I. Nikolaidis, A. Miguel-Cruz, and A. Rios Rincon, \"smart home and home health monitoring technology for the elderly: a systematic review\", International Journal of Medical Informatics, Vol. 91, pp. 44-59, 2016. A. Jacobsson, M. Boldt, and B. Carlsson, risk analysis of intelligent home automation systems, Next Generation Computer Systems, Volume 156, pp. 719-733, 2015. C. D. Kidd, R. Orr, G. D. Abowd et al. 1999 Heidelberg Springer, Berlin, home of perception: Living Laboratory at the International Symposium on Ubiquitous Computing Research. R. Khosla, M. T. Chu, R. Kachouie, K. Yamada, F. Gihiro, T. Yamaguchi, \"Interactive multimodal social robot for improving the quality of care for elderly people in nursing homes in Australia\", in the minutes of the 20 ACM Multimedia International Conference ---MM \'12, pp. 1173- 1176, New York, New York, United States, October-November 2012. N. K. Suryadevara and S. C. IEEE hopadhyay, \"home monitoring system for elderly health measurement based on wireless sensor networks\", IEEE Journal of sensors, Vol. 12, no. 6, pp. 1965-1972, 2012. B. Sathiyabama and S. Malarkkan, \"Low power add-on for MAC units using dual supply voltage in DSP processors\", International Journal of Computer Science and Information Technology, Vol. 32, p. 67, 2012. Z. Luo, Z. Liu, J. Zhang, and C. Song, in the record of the International Conference on measurement, information and control in 2012, \"the design of musical instrument tuning system based on MCU 103 microcomputer\", pp. 79- May 2012, Harbin, China. Y. N. Zhang, H. Y. Ning, J. Bai, B. -C. Chen, P. -C. Zhou, and X. -L. Zhao: \"The elderly are safe early Early warning system based on android mobile phone \"the 10th International Natural Computing Conference in 2014 (ICNC), pp. 1126- 1130, Xiamen, China, August 2014. C. F. Lai, Y. M. Huang, J. H. Park, and H. C. Zhao, \"Analysis of adaptive body posture of the elderly -- Multi-sensor drop detection, IEEE Intelligent System, volume. 25, no. 2, pp. 20-30, 2010. S. H. Kim and D. W. Jin, \"a study on reality Time drop detection system using acceleration sensors and tilt sensors10, no. 5, pp. 1302-1307, 2012. N. Buch, S. A. Velastin and J. A. Orwell, review of computer vision technology for urban traffic analysis, IEEE Intelligent Transportation System trading, Volume 112, no. 3, pp. 920-939, 2011. K. Bahadir and K. Serdar, \"moving target detection and tracking using annealing background subtraction in video: Performance Optimization\", expert system with application, Volume 139, no. 1, pp. 33-43, 2012. M. Cheng and J. Gao, an improved background modeling method for target detection, Springer, Heidelberg, Berlin, 2012. P. Gorur and B. Amrutur, \"accelerated Gaussian mixture model algorithm for background subtraction\", IEEE International Conference on Advanced Video and signal monitoring, 2011 (AVSS), pp. 386- 391, klagenfu, Austria, August-September 2011. S. Gupte, O. Masoud, R. F. K. Martin, and N. P. Papanikoolopoulos, detection and classification of vehicles, IEEE Intelligent Transportation System trading, Volume 13, no. 1, pp. 37-47, 2002. A. Yoneyama, C. H. Yeh, and C. C. J. Guo, \"moving shadow elimination for robust vehicle extraction based on 2D joint vehicle/Shadow model\", in IEEE advanced video and signal-based monitoring meeting notes, P. 2003 229, Miami, Florida, United States, July 2003. G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. About removing shadows from images, IEEE deals on Pattern Analysis and Machine Intelligence, Volume 128, no. 1, pp. 59-68, 2006. A. Leone, C. Distante, N. Ancona, E. Stella, and P. Siciliano, \"texture analysis of shadow elimination in video surveillance systems\", IEEE International Conference on Systems, humans and control theory, 2004 (IEEE Cat. No. 04CH37583), pp. 6325- 6330, Hague, Netherlands, October 2004. R. W. Rahmat, Z. H. Al-Tairi, M. I. Saripan, and P. S. Suleiman, \"hand segmentation shadow elimination based on background subtraction\", International Conference on Advanced Computer Science applications and technology, 2012 (ACSAT), pp. 481- Kuala lumpur, Malaysia, November. J. Yoo, L. Yan, S. Lee et al. , \"A wearable ECG acquisition system with compact plane Stylish circuit board Shirt-based, \"IEEE biomedical information technology deal, Volume 113, no. 6, pp. 897-902, 2009. Zhiwei He (iD) Yang Yu Xiangdong Wai Road (iD) And Gao Mingyu (iD) School of Electronic Information, Hangzhou University of Electronic Science and Technology, Hangzhou, China Communications should go to he Zhiwei; zwhe@hdu. edu. Cn received on February 20, 2017; Revised in May 14, 2017; Accepted on November 21, 2017; Academic Editor Published in January 15, 2018: Yang Yong\'s title: Figure 1: Schematic diagram of the system. Description: Figure 2: Schematic diagram of hardware platform. Description: Figure 3: Schematic diagram of system software. Description: Figure 4: software framework for main control board. Description: Figure 5: Software Flow of main control board. Description: Figure 6: software framework of information acquisition board. Description: Figure 7: flow chart of client software. Description: Figure 9: foreground object detection. Description: Figure 10: experimental results of shadow removal. Description: Figure 11: Sketch of the smallest external rectangle. Description: Figure 12: minimum external rectangle in descending rectangle-down state. Title: Figure 13: an example of getting the smallest external rectangle. Description: Figure 14: The system of the design and its settings.