A step-by-step guide to the use of the Intel OpenCV 1 library

Processing a video sequence

Processing an image sequence is relatively simple using the OpenCV video library. The first step is to specify the filename of the video sequence to be processed:
  
CvCapture* capture = cvCaptureFromAVI(filename);
If a camera is used instead of a video, then function cvCreateCameraCapture should be called. Once this done, you can process each frame by calling the cvQueryFrame function that is, in fact made of the two following calls:
  
if(cvGrabFrame(capture))
      IplImage* img=cvRetrieveFrame(capture);          
Note that the image thus obtained should not be released. At the end of the processing you simply call the releasing function:
  
cvReleaseCapture(&capture);
This is as simple as this. However, in practice, you will want to run the video processing function as a separate thread otherwise your application will hang until the processing terminates. By creating a thread you will be able to interact with the user interface and eventually stop, pause and resume the video processing. This is what we will do now.

But first, when you build an application with more than one thread, you must protect your code against multiple accesses. That is to say that when the data of a class is manipulated by different threads, we must make sure that each manipulation is done by one thread at a time. The standard way to ensure this protection is to use a mutex. Under MS Visual C++, this can be achieved using the WaitForSingleObject function. First the mutex must be created:

  
HANDLE mut;
mut= CreateMutex(NULL, FALSE, NULL);
Then, for each portion you want to protect, you enclosed the block of statement by the following two statements:
  
WaitForSingleObject(mut, INFINITE);
delay= d;
ReleaseMutex(mut);
You then have the guarantee that the enclose portion of code will be executed by one and only one thread at a time. You must be aware, however, that the use of mutex can slow down your code. The mutex must be released when it is no longer in use:
   
ReleaseMutex(mut);
Lets now see how a new thread can be created. This one is started as follows:
   
HANDLE hThread =(HANDLE)_beginthreadex( NULL, 0, processImages,
                 reinterpret_cast<LPVOID>(this), 0, &threadID );
The third parameter is the pointer to the function that will be called as a new thread (here processImages). The fourth parameter is an argument that is sent to the function; it must be a void pointer. Usually you send a pointer to an object containing the data that the function requires. Remember that the call to _beginthreadex starts a new thread; therefore it is a non-blocking method. The execution continues to the next statements concurrently with the execution of the function given in argument. When this later function terminates, it is recommended to end it with:
   
_endthreadex( 0 );
Finally, do not forget to release the thread handle to delete the thread:
   
CloseHandle( hThread );
We can then now present our class that will be responsible for the management of the video stream. This method will be responsible for the creation of the thread that will read each frame of the sequence. The callback function that will be called for each frame is specified through the setImageProcess method while the name of the video file is specified using the setFile method. Here is the signature of the different methods of this class:
   
class AviProcessor {
 
      .    
      .
      .   
 
  public:
 
        // Constructor
        AviProcessor();
 
        // Destructor
        ~AviProcessor();
 
        // set the callback function called for each frame
        void setImageProcess(void (*p)(IplImage *));
 
        // the getter and setters to determine if you want the
        // callback to be called
        void callProcess();
        void dontCallProcess();
        bool callingProcess();
 
        // the getter and setter to determine if you want to display
        // the processed frames
        void setDisplay(bool b);
        bool isDisplayed();
 
        // the getter and setter to introduce a delay between frame
        void setDelay(DWORD d);
        DWORD getDelay();
 
        // a count is kept of the current frame number
        long getFrameNumber();
 
        // size of the frames of the sequence
        CvSize getFrameSize();
 
        // set the name of the video file
        void setFile(char *filename);
        void setFile(CString filename);
 
        // to grad (and process) only one frame
        bool grabOneFrame();
 
        // to grab (and process) the frame sequence
        bool run();
 
        // to restart with a new video
        bool restart();
 
        // to stop the capture
        void stopIt();
        bool isStopped();
};
The numerous setters and getters allow controlling the behavior of the class. For example, you may want the current frame to be displayed or not (setDisplay) or to be processed or not (callProcess, dontCallProcess) . The run method is the one that creates the thread; it is simply defined as:
    
        bool run() {
 
          // make sure that a thread is not already running
          if (!isStopped())
                  return false;
 
          // destroy any previoulsy created thread
          if (hThread != 0)
             CloseHandle( hThread );
 
          stop= false;
 
          // start the thread
          hThread = (HANDLE)_beginthreadex( NULL, 0,
                  processImages,reinterpret_cast<LPVOID>(this), 
                  0, &threadID );
 
          return true;
        }
The new thread starts the execution of the function processImages (declared as a friend of class AviProcessor). This function is defined as follows:
     
unsigned __stdcall processImages( void *g) {
 
      AviProcessor *proc= reinterpret_cast<AviProcessor *>(g);
 
      while (!proc->isStopped()) {
 
            if(cvGrabFrame(proc->capture)) {
 
                  proc->img=cvRetrieveFrame(proc->capture);          
 
                  // calling the process function
                  if (proc->callingProcess())
                        proc->process(proc->img);
                  else
                        Sleep(proc->getDelay());
 
                  // displays image and frame number
                  if (proc->isDisplayed()) {
 
                        CvScalar sc;
                        char text[50];
                        sprintf(text,"Frame #%6d",
                                     proc->getFrameNumber());
 
                        sc= cvGet2D(proc->img,20,20);
                        if (sc.val[1]>128)
                              cvPutText(proc->img, text, 
                                        cvPoint(20,20), 
                                        &(proc->font), 
                                        cvScalar(0,0,0));
                        else
                              cvPutText(proc->img, text, 
                                        cvPoint(20,20), 
                                        &(proc->font), 
                                        cvScalar(255,255,255));
 
 
                        cvShowImage( "Image (AviProcessor)", 
                                     proc->img );
                        HWND hWnd= (HWND)cvGetWindowHandle(
                                     "Image (AviProcessor)");
                        // force the window to repaint
                        ::SendMessage(hWnd,WM_PAINT,NULL,NULL); 
                  }
 
                  // increment frame number
                  proc->incFrameNumber();
 
            } else {
 
                  proc->stopIt();
            }
      }
 
    _endthreadex( 0 );
      return 0;
}
Essentially, the function is a loop that grabs and retrieves the video frame (cvGrabFrame, cvRetrieveFrame) until the process is stopped or no more frame remains. For each frame the processing function is called (proc->process(proc->img)). A series of statement display the current frame and overlay the frame number onto it. There is also a very similar method in AviProcessor that captures only one frame and then returns; this is especially useful when you need to know the size of the frames in the video before starting the video processing. This method is as follows:
     
       bool grabOneFrame() {
 
          // make sure that a thread is not already running
          if (!isStopped())
                  return false;
 
            if(cvGrabFrame(capture)) {
 
                  // gets the current image
                  img=cvRetrieveFrame(capture);          
 
                  // calling the process function
                  if (callingProcess())
                        process(img);
 
                  if (display) {
 
                        .
                        .
                        .
                  }
 
                  // increments current frame number
                  incFrameNumber();
            }
 
            return true;
        }
As it can be seen, it is essentially the same as before but without the loop. Finally, the stopIt method stops the running thread by setting the stop variable to true. This is the classic way of stopping a thread; it ensures that this one can finish a complete loop turn before stopping nicely.
     
       void stopIt() {
 
          WaitForSingleObject(mut, INFINITE);
              stop= true;
          ReleaseMutex(mut);
        }
It is also possible to restart a new processing thread by releasing the previous CvCapture object and creating a new one:
      
        bool restart() {
 
          // make sure that a thread is not already running
          if (!isStopped())
                  return false;
 
            // destroy any previoulsy created thread
            if (hThread != 0)
             CloseHandle( hThread );
 
            fnumber=0;
            stop= false;
          cvReleaseCapture(&capture);
            capture = cvCaptureFromAVI(aviName.GetBuffer());
 
            return true;
        }
If you call run instead, the processing will resume at the point where it stops previously.

Now to create an application using this video processing class, we will use a controller. Again, we illustrate the video processing using the Sobel class. The controller will then creates the AviProcessor and Sobel classes.

       
      SobelVideoController() { // private constructor
 
              //setting up the application
              aviproc= new AviProcessor();
              sobel= new Sobel();
              aviproc->setImageProcess(process);
              aviproc->callProcess();
              aviproc->setDisplay(true);
      }
All controls we need are the ones who would allow us to specified the video filename, start and stop the processing.
       
class SobelVideoController {
 
  private:
 
      static SobelVideoController *singleton; // ptr to the singleton
 
      AviProcessor *aviproc;
      Sobel *sobel;
 
        void run() {
 
              aviproc->run();
        }
 
        void stop() {
 
              aviproc->stopIt();
        }
 
        inline void setFile(CString filename) {
 
              aviproc->setFile(filename);
        }
 
      .
      .
      .
We also add two methods that will be used in the callback: one that performs the processing (i.e. call the appropriate method in Sobel) and the other to obtain the current frame:
        
        inline void processImage(IplImage *image) {
 
              sobel->processImage(image);
        }
       
        inline IplImage* getOutputImage() {
 
              return sobel->getOutputImage();
        }
The callback is then defined as follows:
        
void process(IplImage* img) {
 
      SobelVideoController::getInstance()->processImage(img);
      cvShowImage( "Result",
                  SobelVideoController::getInstance()
                                    ->getOutputImage());
      HWND hWnd= (HWND)cvGetWindowHandle("Result");
      UpdateWindow(hWnd); // force the window to repaint
}
Now, the user interface is simply the following:

With this architecture the two main buttons (Process and Start) are trivially define as:
         
void CcvisionDlg::OnProcess()
{
      SobelVideoController::getInstance()->run();
}
 
void CcvisionDlg::OnStop()
{
      SobelVideoController::getInstance()->stop();
}

Check point #9: source code of the above example.

Top of the page

 

(c) Robert Laganiere 2011