WP1 – Experimentation and establishment of technical solutions
This section presents the main results for the activities that took place in WP1 – Experimenting and establishing technical solutions of the SoV Lite project.
Results:
(1) Report on the selected equipment and algorithms
In this WP we studied the field of assistive systems, identifying the latest methods of image processing for navigation and understanding of the environment, as well as the latest video cameras that provide depth information. Preliminary experiments were performed with the candidate equipment and the most suitable solutions for the 3D pipeline were identified. Given the constraints of the SoV Lite project on processing acquisition units, the best solutions for safe navigation and spatial understanding of the traversed environment are based on the reconstruction of space through a SLAM (simultaneous localization and mapping) technique that uses a representation of space either by a cloud of sparse (scattered) dots or by individual objects (characterized by framing boxes or dots representing the centers of gravity of the framing boxes).
During this stage we conducted a review and a comparative analysis of the solutions for the semantic description of the scene, identifying possible advantages and disadvantages of these methods in terms of the requirements of an assistance system dedicated to the blind.
(2) Report on experiments with new hardware
Following the preliminary experiments, the ZED stereo camera was selected for color and depth imaging and the Jetson processing unit. A comparative analysis of ZED 2 – ZED mini and Jetson Nano – Jetson Xavier NX concluded the opportunity to use ZED 2 and Jetson Xavier NX in the new architecture. The key advantage of ZED 2 is the ability to use the built-in AI module in the SDK, while the Jetson Xavier NX benefits from significantly superior resources and the built-in WiFi / BT module in physical size and consumption comparable to Jetson Nano.
The ZED 2 camera was integrated into the SoV Lite system, and the aquisitioned sequences could be processed using the modules from the old system (Sound of Vision). In order to validate the prototype architecture, semantic segmentation tests and object detection were performed on the Nvidia Jetson platform.
(3) Report on experiments and preliminary implementations of computer vision modules databases with annotated video recordings
In WP1, experiments were also performed with algorithms for spatial understanding of the environment, for how perception and navigation work. Possible techniques that can be included in the SoV Lite pipeline have been identified, taking into account the constraints imposed by the project requirements. Among them, two promising solutions proved to be promising. On the one hand we can use the ZED SDK (which has support for detecting people and machines in images) in combination with TensorFlow (to be able to use our own data sets). On the other hand, YOLOv3 proved to be promising.
During this stage we selected and evaluated a series of existing solutions in the literature for the semantic description of the scene. We identified two types of methods, based either on object detection and recognition, or on semantic segmentation. The experiments show that both the running time and the accuracy of the selected methods depend directly on the resolution of the images. Also, based on the above observations, we identified a possible solution to improve the inference time while maintaining the accuracy of the results.
(4) System specifications document
During this stage, the actors and the need for hardware equipment were identified, and the functional and non-functional requirements of the system were established. The administrator manages subscriptions, games and views reports about SoV Lite subscribers. The researcher analyzes the information obtained after testing the system. It has access to a selection of data from system testing sessions and can view statistics and graphs. The visually impaired people (VIP) are the main beneficiaries of the system. They have access to system operating mode settings (scanning distance and aperture, playback mode, etc.). They can use the “Browse and playback of the environment” functionality, the default mode of operation of the system. VIP can also use the functionality of semantic scene description and text identification and reading. A very useful feature is that of training in virtual environments and playing some games. VIP also have access to features for purchasing subscriptions and making system software updates.
System specifications defined with use cases
(5) Architectural design document
The architectural design of the SoV Lite system is based on the description of the functional and non-functional requirements set out in the system specifications. Based on this, a series of design objectives have been established that will be taken into account in the detailed implementation of the software architecture. An architectural scheme has also been defined, which will be the basis for the implementation of the hardware-software system.
UML diagram of the SoV system that includes all the hardware and software modules of the system, including the server services part
(6) 2 review articles
A Brief Classification of Computer Vision-Based Solutions for the Visually Impaired
Andrei Lăpușteanu, Otilia Zvorișteanu, Anca Morar, Alin Moldoveanu, Florica Moldoveanu, Simona Caraiman, Victor Asavei, Ștefan Achirei
On improving perception for visually impaired:requirements, research and practicality
Otilia Zvorișteanu, Ștefan Daniel Achirei, Nicolae Alexandru Botezatu, Robert Gabriel Lupu, Adrian Burlacu and Simona Caraiman – e-Health and Bioengineering (EHB) 2020.
WP2 – Prototype implementation and validation
This section presents the main results for the activities that took place during WP2 – Implementation and prototype validation of the SoV Lite project.
Results:
(1) Product: Initial prototype
(2) Product: Mobile application for system control
(3) Product: Mobile application for visually impared people training
(4) Product: Final prototype (partial)
(5) User manual (partial)
Deliverables:
(1) Description of the initial prototype
At this stage, the first fully functional prototype (called the initial prototype) of the SoV Lite system was made, a naturalistic, accessible and ergonomic audio-haptic sensory substitution for the VIP. The system comprises, from a hardware point of view, a video aquisition system, a control and processing system and an audio playback system, and from a software point of view, a central control and processing component, a perception and navigation component and a semantic description component. The system is able to understand the scanned environment and assist the user in navigating and understanding the environment – by identifying surrounding objects and potential hazards and encoding them in sound and haptic information.
(2) Assessment of technical capabilities
Several tests, both hardware and software, were defined and performed to verify and evaluate the functionality of the components of the initial SoV Lite prototype. Following this evaluation and testing process, a first set of necessary improvements and technical corrections were extracted.
(3) Report on prototype testing in laboratory environments and real conditions
Several usage scenarios have been defined and implemented to ensure the correct operation of the system in as many cases as possible, as well as the accessibility of the components of the initial SoV Lite prototype. Following this evaluation and testing process, a number of new improvements have been made to both the functionality and the user interface, adapted for the visually impaired.
(4) Technical description of the final prototype (partial)
At this stage, the realization of the complete prototype was started, called the final prototype of the SoV Lite computer system. To achieve this, the systems of acquisition, processing, perception, navigation and semantic description have been considerably improved following the tests performed using the initial prototype. A mobile application for training in virtual environments has also been created.
(5) Report on blind training techniques
User-centered training and instructional protocols and procedures have been defined that are clear, flexible, and adaptable to the progress of each individual. The training program follows a gradual approach on three levels of difficulty: in the virtual environment, in the real world, indoors and in the real world, outdoors.
(6) Report on final technical capabilities (partial)
Several hardware and software tests were defined and performed to verify and evaluate the functionality of the components of the final SoV Lite prototype. Following this evaluation and testing process, a number of new improvements were extracted.
(7) Final prototype test report (partial)
Several usage scenarios have been defined and implemented to ensure the correct operation of the system in as many cases as possible, as well as the accessibility of the components of the final SoV Lite prototype. Following this evaluation and testing process, a number of new improvements were extracted.
(8) User manual (partial)
A detailed user manual has been developed which includes details of the installation, maintenance and use of the SoV Lite system and the training application. A number of necessary safety precautions have also been developed and must be followed by direct users and their carers before, during and after use of the device.
(9) 3 articles (presentation of system functions and architecture, final navigation / perception solutions, test results)
Pothole Detection for Visually Impaired Assistance
Ștefan Daniel Achirei, Ioana Ariana Opariuc, Otilia Zvoristeanu, Simona Caraiman and Vasile Ion Manta – 2021 International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, România: 2021
Environment Understanding Framework for Intelligent Mobile Systems
Otilia Zvoristeanu, Ștefan Daniel Achirei, Simona Caraiman, and Vasile Ion Manta – 25th International Conference on System Theory, Control and Computing (ICSTCC), Iasi, Romania: IEEE, 2021 10.1109/ICSTCC52150.2021.9607078
Otilia Zvorișteanu, Simona Caraiman, Robert-Gabriel Lupu, Nicolae Alexandru Botezatu, Adrian Burlacu – Electronics 2021. https://doi.org/10.3390/electronics10141619
(10) Website update
WP3 – Transition preparation and continuation (en)
This section presents the main results for the activities that took place within WP3 – Transition preparation and continuation. As the project has started later due to reasons that were not related to us, the reported period covers the period January – May 2022, and thus represents a period of 5 months out of the 24 forecasted of the project (months L20 – L24).
During the current stage, a part of the activities started within WP2 were continued and completed, the development activity being completed.
Based on an initial prototype that was tested in laboratory environments and real-world conditions, new requirements, optimizations, and operating or operability issues were extracted. These results were used to considerably improve the systems of acquisition, processing, perception, navigation and semantic description, thus achieving the final prototype that was tested and piloted with blind people.
The final prototype ensures a complete functionality, according to the proposed objectives, and has a high degree of autonomous ease of use, validated at TRL 6 level, thus being very close to the needs for a commercial finished product. In order to use the system, the training techniques were adjusted to accommodate the new system, introducing individual or remote training techniques, the user manual was finalized and the mobile application for training in virtual gaming environments (VTE – virtual training environment) for training of blind people with the audio / haptic playback system of the SoV Lite system was improved and finalized. The test results generally provided positive feedback, but also practical use important observations, especially to ensure a greater accessibility and utility of the system.
Also, within the project were developed a series of experimental graphical game-like applications, which can be used by users as a means of relaxation or entertainment. This area is little explored and novel for blind people, who were pleasantly impressed and very receptive. Thus, in this project we defined and implemented the bases of a platform for the distribution of interactive graphic applications or games dedicated to people with visual impairments, a truly novel and innovative solution.
A plan for further scientific and technical improvements and commercialisation of the project was also developed.
The dissemination activities were carried out according to plan and consisted of publishing 2 scientific articles, and updating the website.
In conclusion, all the objectives of phase 3 and the project were met, reaching a TRL-6 maturity level.
At the time of project completion, we can report the following results obtained during the current implementation phase:
Results obtained:
(1) Product: SoV Lite Final Prototype
(2) Product: Mobile application for control and audio playback
(3) Product: Mobile application for training people with visual impairments
(5) User’s Manual
Deliverables obtained:
(1) Initial prototype
At this stage, the first fully functional prototype, called the initial prototype of the SoV Lite information system, of naturalistic audio-haptic sensory substitution, accessible and ergonomic for the blind, was completed and tested. The system comprises, from a hardware point of view, a aquisition system, a control and processing system and an audio playback system, and from a software point of view a central component of control and processing, a component of perception and navigation and one of semantic description. The system is able to understand the scanned environment and assist the user in safe navigation and in better understanding of the environment by rendering information, identifying surrounding objects and especially dangers and coding them in sound information.
(2) Techniques for training the blind
We have completed and finalized a list of protocols and procedures for user-centered training and testing, that are clear, flexible and adaptable to the progress of each individual. The training program follows a gradual approach on three levels of difficulty: in the virtual environment, in the real world, indoors and in the real world, outdoors.
(3) The final prototype
In this WP the development of the complete prototype, called the final prototype of the SoV Lite information system, was completed. In order to achieve it, the systems of acquisition, processing, perception, navigation and semantic description have been considerably improved following the tests carried out using the initial prototype. A mobile app has also been created for training in virtual environments.
(4) Evaluation of final capabilities
Several hardware and software tests have been defined and carried out to verify and evaluate the functionality of the components of the final SoV Lite prototype. As a result of this evaluation and testing process, a new set of necessary improvements have been extracted and then developed in an iterative approach.
(5) Testing of the final prototype
Several use scenarios have been defined and carried out to ensure the correct operation of the system in as many cases as possible, as well as the accessibility of the components of the final SoV Lite prototype. A number of improvements have been extracted from this evaluation and testing process and then integrated in an iterative approach.
(6) User’s manual
The elaboration of a detailed user manual that includes both details of the installation, maintenance and use of the SoV Lite system and the training application has been completed. A number of necessary safety measures have also been developed that direct users and their carers must comply with before, during and after using the device.
(7) Technical plan for the continuation of the project
The SoV Lite system has reached an advanced stage of prototyping (TRL-6) and can be ready for commercialization. In this regard, plans were made to improve the technical development, to prepare for the launch of a commercial version and to commercially exploit the result.
(8) Dissemination
2 articles:
| Gradient-based image segmentation for inadequately lit or textured environments
Andrei Laptusteanu, Florica Moldoveanu, Anca Morar, Alin Moldoveanu, Maria Anca Balutoiu, Victor Asavei – 2022 Tenth International Workshop on Assistive Computer Vision and Robotics (ACVR), European Conference on Computer Vision (ECCV) 2022 |
| SoV Lite – More Steps Towards Affordable Visual to Audio-Haptic Sensory Substitution
Anca Morar, Alexandru Gradinaru, Maria-Anca Băluţoiu, Ştefan Achirei, Otilia Zvorişteanu, Alin Moldoveanu, Florica Moldoveanu, Simona Caraiman, Andrei Lăpuşteanu, Gabriel Ivănică, Robert Lupu, Computer Vision and Image Understanding Journal, Elsevier, ISSN: 1077-3142 |
Also, other important forms of project dissemination were carried out, namely:
- Establishing contacts in the press; preliminary presentations of the project objectives to the contact points. They will allow the efficient dissemination of the final results, through televisions, radio and online press and support for exploitation;
- Establishing contacts with a large number of therapists and preliminary presentations of the project objectives. They will then allow support for exploitation;
- Student research topics at master level, finalized with dissertation papers, in UPB, Faculty of Automatic Control and Computers.
- PhD topics at UPB in the field of games for the blind (Silviu Ivașcu) + PhD at TUI Otilia Zvorișteanu (“Environment perception and understanding for intelligent mobile systems”)
(9) Website update