Cléopatre: Real-Time Vision

Overview

Shelves

Publications

Cléopatre Vision: Technical Report 2005 Technical Report for the Real-Time Vision part of Cléopatre project.

PhD Thesis 2005 Dynamically Reconfigurable Robust Real-Time Vision System for Mobile Robotics

RoboCup 2005 New Video Rate Region Color Segmentation and Classification for Sony Legged RoboCup Application

ISSPA 2005 Clear Box Evaluation of Vision Algorithms Application to the Design of a new Color Region Growing Segmentation for Robotics

SPIE 2005 Video Rate Color Region Segmentation for Mobile Robotic Applications

SPIE 2005 Temporally Optimized Edge Segmentation for Mobile Robotics Applications

MVA 2005 Fast Quality Measurement of a H263+ Video Stream for Teleoperating a HRP-2 Humanoid Robot

JFAAA 2004 Algorithme d'ordonnancement de la bibliothèque open-source Cléopatre : application à la réalisation d'un système de vision temps-réel dynamiquement reconfigurable.

RSJ 2004 New Image Quality Measurement of a H263+ Video Stream for Teleoperation

ISSPA 2003 Towards a Method to Compare and to Evaluate Fast Pixel Gathering Mechanisms for Real Time Robotic Vision Systems

GRETSI 2003 Vers une Méthode d'Evaluation Comparée de Mécanismes de Regroupements de Pixels : Application à la Décomposition en Composantes Connexes et à la Croissance de Régions pour l'utilisation dans un Système de Vision Robotique Temps Réel

Videos and Sequences
	Pekee, straight to boxes: avi, mpeg4, sequence. Pekee is running straightforward on a tiled floor in a place full of boxes, that are quite geometric shapes in background. The purpose of this sequence is to study the apparent optical flow measured on the floor, and to segment the view in order to localize the possible obstacles. As this sequence has been recorded with a wireless video camera in a place with several magnetic disturbance sources, interferences can be seen at the top of the frames.
	Pekee, straight to wall: avi , mpeg4 , sequence. In this sequence, Pekee is running straightforward in to a wall. This wall is granulous and has apparent bricks. The purpose of this sequence is to see if optical flow or segmentation operators are able to detect that this type of surface is an obstacle coming closer.
	Pekee, turning camera off: avi , mpeg4 , sequence. In this sequence, the camera is suddenly unplugged. The pupose of such a sequence is to check that a vision system is able to detect an anomaly and to send warning messages to the higher level management operator. With such kind of noisy image, there should not be any regions or outlines detected.
	Simulator, closed-loop commanded through FIFO: avi, mpeg4, sequence. For this sequence, the simulated robot is commanded through FIFO by an external application, and sends back odometry measures to this controller. This sequence has been made to have a sequence representative of a robot evolving in an industrial environment and avoiding obstacles.
	Simulator, closed-loop commanded with monospectral lights: avi, mpeg4, sequence. This sequence is the same as the previous one, except that 2 monospectral (red and blue) and a white light have been set. It generates an inhomogeneous lighting that can be perceived by human eye (which is more difficult with three white lights).
	Simulator, still robot with variable lighting: avi, mpeg4, sequence. In this sequence, the robot does not move, and there is a single light commanded through FIFO by an external application, in order intensity to variate pseudo-sinusoidally. This allows to see results of various segmentation operators when lighting conditions are not constant.
	Simulator, collision with variable lighting: avi, mpeg4, sequence. For this sequence, the simulated robot is going straightforward and will collide a box. At the same time, light is decreasing. The purpose of this sequence is to check how much this hinders the collision detection.
	Simulator, single attenuated light: avi, mpeg4, sequence. In this sequence, the robot is going straightforward in a corridor without obstacles. The light is not commanded, but its attenuation parameter is set so intensity is decreasing when the robot is moving away the light source.
	Mobile object commanded in open-loop, rotating.: avi, mpeg4, sequence. In this sequence, a textured mobile octaedron is commanded in open-loop by an external application in order to have a rotative movement. Texture has been set by the user. The interest of this sequence is to display the optical flow generated by this particular texture for this particular movement.
	Mobile object commanded in open-loop, translating.: avi, mpeg4, sequence. In this sequence, a textured mobile cube is commanded in open-loop by an external application in order to be translated. The interest is the same as above: detect a mobile object and determine its movement.
	Two mobiles objects, rear flares moving away: avi, mpeg4, sequence. As up to four independent mobiles object can be used at the same time, this sequence shows two red cubes moving away. This is used to simulate rear flares of a vehicle (e.g. another robotic conveyor), and to determine its distance with a color region segmentation (knowning spacing between flares).

Detailed Content

cleo.visionAlgorithms

cleo.visionAlgorithms package is a collection of temporally-optimized vision processing algorithms, dedicated for very constrained robotic applications where the size of the generated code must be reduced as much as possible. For this, each algorithm is provided in an independant file that can be compiled separately.
This collection gathers several families of algorithms:

• generic image management functions
• mathematical morphology operators
• edge detection operators
• region segmentation operators
• connexe components decomposition operators
• color classification operators
• edge thinning and prolongation operators

cleo.visionToolkit

• frames management,
• mathematical morphology,
• region segmentation,
• connexe components decomposition
• color classification
• edges detection
• edges thinning and prolongation
• thresholding
• optical flow computation
• pin-diagram display
• image format conversion
• boxes management

• cleo.framegrabber, that grabs periodically frames from a video device, modify its format if required, and sends them through UDP or TCP sockets.
• cleo.listener, that gets frames from a UDP or TCP socket, and sends them through FIFO, or stores them into a shared memory area or a file.
• interceptor, a graphical user interface to inspect simultaneously outputs of several operators of a vision processing chain running in user's space or kernel space. This tool enable to see repercussions of a modified parameter on the whole chain in a single glance.
• Testing programs that can also be taken as models to learn how to use Cléopatre's Vision library.
• Benchmarking programs to check performances of the operators on one's machine.
• Small tools to display and analyze benchmarks results.

cleo.visionSimulator

• Simulation of a holonomic mobile robot equipped with a robotic head and from one to three cameras. The head has pan/tilt/convergence axes, and each camera can also has the three rotation axes of freedom. Optical axes can also be displayed.
• Highly parametrable display: the more numerous views are, the slower the simulator will be. To prevent useless views to be drawn, a command line options allows to select only required views to fit the best user's needs. Dimensions of views can also be set.
• Up to five views: left, right camera and upper, plus a bird's view and a global view to have a better sight of the robot within its environment.
• Video rate frame grabbing: as the simulator has been designed to be substituted to a real video acquisition device, video rate can be 25 Hertz with a binocular system. Frames are set with Cléopatre format, i.e. their header is filled with several informations: timestamp, number of frame, origin and so on ... Besides, acquisition period can be set by user.
• Frames transmission: grabbed frames can be transmitted to cleo.listener through network socket; the simulator can be transparently substituted to cleo.framegrabber application.
• Several ways to command robot: the robot can be moved by a human operator using keyboard or joystick. It can also be commanded in open-loop using a FIFO.
• Odometry and closed-loop command: each freedom axis can send odometry informations through FIFO. This enable closed-loop command of the robot.
• Highly parametrizable lighting, with up to 8 independant lights, which properties can be set when launching the program, or commanded through FIFO afterwards.
• Collision detection: collision between the robot and the building are detected, so the robot cannot go through walls as in reality.
• Static and mobile obstacles: several static obstacles can be set into this environment, to predefined places or user defined ones. Also, it is possible to command special obstacles in open loop through FIFO, with 6 degrees of freedom (3 translations, 3 rotations). All these items can be textured easily by the user without recompiling the application (use of symbolic links).
• Color customizable landmarks, that can be enable or not at launch time. Color of their strips can be set by the user.

• Textures realism: to be able to test edge detection, region segmentation and optical flow operators, textures have been created from snapshots of real elements, as walls or floor.
• Lighting realism: e.g. it is possible to have non-uniform lighting conditions by using various sources, and setting their attenuation parameters. It is also possible to have dynamic lighting to simulate a suddenly turned off light, or some other light effects as the 50 Hertz influence (sinusoidal command) or a defectuous neon tube.
• Robotic realism: the axes of freedom correspond to the robotic conveyor, demonstrator for the Cléopatre project. The axes of the head and of the cameras correspond to those found in several robotics devices.
• Video realism: the video rate acquisition of 25 Hertz is possible with a binocular system and a recent computer. The color format of the frame can be set to RGB or grey scale. The size of the frame can be SQCIF, QSIF, QCIF, SIF or CIF.
• Scenarii possibility: with open or close loop of the various elements, it is possible to create several situations to test specific things, as obstacle avoidance, target following in case of occultation, capture of a video sequence for a specific route when the robot is commanded by a sequencial external application through FIFO, and so on ...

PredN

During Cléopatre Project, following developments have been done:

• It is possible to create objects from the hierarchical composition of other objects.
• Scheduling model has been extended. Before, the model didn't allow to run more than one function during the operational part of an object instance. This limitation was intended to be able to take into account Imprecise Computation and Back-Up procedures. It is now possible to specify several functions.
• Two plug-ins for RT-Cléopatre code generation have been implemented. These plug-ins correspond to versions released in July and December 2003.
  PredN was beta-prototype at the beginning of the project; to make it more accessible and pertinent within Cléopatre context, a significant revision and an entire debugging have been carried out.

quality

• Ffmpeg (http://ffmpeg.sourceforge.net) is an efficient implementation of the H263+ protocol (amongst others like MPEG-4, H263, H264 ... ). This protocol is used in whole world for numerous commercial products (Real Video streaming, France Telecom, ... ) and free projects (Media Player, VLC, ... ).
• VLC (VideoLAN Client, http://www.videolan.org) is used to get frames from the camera, to apply adapted encoding and decoding algorithms, and to create and read network paquets.
• VLC has been modified in order to use the three quality measures: GBIM, UIQ and EQM. FFmpeg implementation of H263+ protocol has also been studied, and a modification of the encoding process has been proposed in order to control video stream quality.

Experimental platform has been tested with the HRP-2 humanoïd robot during a teleoperation experiment with two configurations: first one was a 100Mbit/s wired network connection, and the second one used a WiFi wireless link.
With the first configuration, the evaluation of quality measures has been completed successfully; with the second one, only one measurement has been made used of. This experiment has been accepted for an international conference (IAPR Cnference on Machine Vision Applications, 16-18 May 2005).

Publications

2005

Technical Report for the Real-Time Vision part of Cléopatre project.

Patrick Bonnin

This document is the Technical Report that concludes the Real-Time Vision part of the Cléopatre project.

Dynamically Reconfigurable Robust Real-Time Vision System for Mobile Robotics

A. de Cabrol

This thesis aims to design a quite generic vision system that can be used in robotics. Therefore, it has to meet requirements of embedded systems: robustness, reliability, safety and still a low cost.
So, such a system must be able to make the best use of its resource, and this is possible from now on owing to innovative mechanisms of the real-time layer of the RNTL Cléopatre project, lead by French Ministry of Science and Technologies.
To define precisely the generic skills we were looking for, we analyzed what kind of tasks that were required by the robotic system could only be accomplished by vision system, and what were the constraints imposed by this robotic context. Taking into account these needs with corresponding operators leads to following conclusion: both generic skills and robustness we are seeking need the most exhaustive information extraction in the image, as edges, color regions and optical flow features are complementary, and also perturbed to different kinds of disturbances. Therefore, we have designed and implemented new more efficient operators that fit with real-time processing requirement. Besides, integration of the vision subsystem into a whole robotic application has leaded to design new suitable means of validation and debugging.
When this base of low-level vision processing has been completed, we deigned a specific architecture, adapted to the topic of this thesis. As the system has to be embedded and must not have to get frozen, we have thought about how to use at best real time applicative layer of Cléopatre to obtain the best algorithm architecture adequation that, even if it has been designed to fit closely the constraints, ensure a safe functioning of the robot. Hence we implemented such a dynamically reconfigurable vision system. By that mean, our proposed architecture has been validated, and we set up a methodology to use efficiently fault tolerance mechanisms of Cléopatre for robotics vision.

A New Video Rate Region Color Segmentation and Classification for Sony Legged RoboCup Application

A. de Cabrol, P.J. Bonnin, T. Costis, V. Hugel, and P. Blazevic

Whereas numerous methods are used for vision systems embedded on robots, only a few use colored region segmentation mainly because of the processing time. In this paper, we propose a real-time (i.e. video rate) color region segmentation followed by a robust color classification and region merging dedicated to various applications such as RoboCup four-legged league or an industrial conveyor wheeled robot. Performances of this algorithm and confrontation with other existing methods are provided.

Clear box evaluation of vision algorithms: application to the design of a new color region growing segmentation for robotics

Aymeric de Cabrol, Patrick J. Bonnin, Maryline Silly-Chetto, Vincent Hugel et Pierre Blazevic

We propose a Clear Box Evaluation Methodology. Its goal is to create new region segmentations from the principles of well known ones.
We illustrated this by an example of creation of a new color segmentation for robotic vision applications. First, we present the evaluation criteria used for these specific applications. For this purpose, four well known algorithms are evaluated. The results of this evaluation are the outlines of the new created segmentation. Then, the new created method is detailed and evaluated. The quality of the obtained segmentation is widely sufficient for robotic applications and its implementation satisfy the real time constraint (i.e. the video rate).

Video Rate Color Region Segmentation for Mobile Robotic Applications

Aymeric de Cabrol, Patrick J. Bonnin, Vincent Hugel, Pierre Blazevic and Maryline Silly-Chetto

Color Region may be an interesting image feature to extract for visual tasks in robotics, such as navigation and obstacle avoidance. But, whereas numerous methods are used for vision systems embedded on robots, only a few use this segmentation mainly because of the processing duration.
In this paper, we propose a new real-time (ie. video rate) color region segmentation followed by a robust color classification and a merging of regions, dedicated to various applications such as RoboCup four-legged league or an industrial conveyor wheeled robot. Performances of this algorithm and confrontation with other methods, in terms of result quality and temporal performances are provided. For better quality results, the obtained speed up is between 2 and 4. For same quality results, the it is up to 10.
We present also the outlines of the Dynamic Vision System of the CLÉOPATRE Project for which this segmentation has been developed and the Clear Box Methodology which allowed us to create the new color region segmentation from the evaluation and the knowledge of other well known segmentations.

Temporally Optimized Edge Segmentation for Mobile Robotics Applications

Aymeric de Cabrol, Patrick J. Bonnin, Vincent Hugel, Kamel Bouchefra and Pierre Blazevic

As an Edge may be an interesting image feature to extract for robotic visual tasks such as the 3D modelization of the environment of robots by stereovision, we propose in this paper a methodology to implement edge segmentation, then we apply it to design a Temporally Optimized Edge Segmentation for Mobile Robotics Applications.
Using our methodology, we show how it is possible to reduce the duration of an edge detection operator from 100.62ms for the slower case to 10.8 ms for the faster one. This represents a gain of nearly 90ms for the processing time, so nearly a factor of 10 for the speed up.

Fast Quality Measurement of a H263+ Video Stream for Teleoperating a HRP-2 Humanoid Robot

O. Stasse, Neo Ee Sian, K. Yokoi, G. Dauphin and P. Bonnin

This paper describes an experimental software platform for measuring image quality on a H263+ video stream. The goal is evaluate the lost of quality induced by the compression scheme. This architecture has been used while teleoperating a humanoid robot. Among the three quality measurements presented, a new distance is proposed to tackle the reference problem.

2004

Algorithme d'ordonnancement de la bibliothèque open-source Cléopatre : application à la réalisation d'un système de vision temps-réel dynamiquement reconfigurable.

A. de Cabrol, P.J. Bonnin, M. Silly-Chetto and T. Garcia

Cet article décrit d'abord les principes d'un système d'exploitation temps réel modulable, ouvert et libre appelé Cléopatre et montre ensuite son utilité pour réaliser un système de vision temps-réel dynamiquement reconfigurable. Ce système d'exploitation temps-réel construit à partir de Linux se présente sous forme de composants logiciels sélectionnables dans une bibliothèque et chargeables dynamiquement. Les composants sont offerts sur quatre étagères : l'ordonnanceur à priorité fixe ou dynamique, la synchronisation par sémaphores avec des protocoles dits à priorité plafond, la gestion des tâches apériodiques avec des serveurs optimaux et la tolérance aux fautes avec le Mécanisme à Echéance ou le Calcul Imprécis. Finalement, nous décrivons comment ces mécanismes peuvent répondre aux exigences de la vision robotique.

New Image Quality Measurement of a H263+ Video Stream for Teleoperation

O. Stasse, G. Dauphin, and P. Bonnin

This paper describes an experimental software plateform to measure image quality on a H263+ video stream. The goal is to measure the lost of quality induced by the compression scheme. The possible application of this platform is to control the encoder according to the quality measurement while teleoperating a mobile robot. Among the three quality measurements presented, a new distance is proposed to tackle the reference problem.

2003

Towards a Method to Compare and to Evaluate Fast Pixel Gathering Mechanisms for Real Time Robotic Vision Systems

P. Bonnin, O. Stasse, V. Hugel, P. Blazevic and G. Dauphin

Taking into account the constraints of mobile and autonomous robotics, and the tasks devoted to the vision system embedded on different robots for various applications, we propose a research in matter of fast pixel gathering mechanisms for the connected component extraction and the color region segmentation.
In this work, we studied three different mechanisms, and proposed a new method to compare and to evaluate the algorithms according to two criteria: the speed of processing and the quality of the results for robotic applications. This method allows to choose the most appropriate algorithm for a given robotic application, and to create new algorithms, by hybridizing older ones while keeping interesting properties.

Vers une Méthode d'Evaluation Comparée de Mécanismes de Regroupements de Pixels : Application à la Décomposition en Composantes Connexes et à la Croissance de Régions pour l'utilisation dans un Système de Vision Robotique Temps Réel

P.J. Bonnin, O. Stasse, V. Hugel, P. Blazevic and M. Silly-Chetto

We propose a method to compare and to evaluate the algorithms of fast pixel gathering mechanisms for the connected component extraction and the color region segmentation. The evaluation is based on two criteria: the speed of processing and the quality of the results for robotic applications.