A DirectShow Tutorial

Processing an image sequence

It is now time to process an image sequence. What we want to do is to sequentially process each frame of an AVI sequence. To do so, the OpenCV library offers a special filter called ProxyTrans. It should be located in C:\Program Files\Intel\opencv\bin. To be used, it must first be registered. This can be done from the MS-Dos window using the regsvr32 application (you just type regsvr32 ProxyTrans.ax, you might have to include C:\Program Files\Intel\opencv\bin in your path environment variable). To check if the ProxyTrans filter is ready to be used, we use again the GraphEdit application. Build a rendering filter graph and then delete the connection between the Decompression filter and the Video Renderer (just click on the arrow and push on Delete button). Now select Select Graph|Insert Filters..., the ProxyTrans should be in the list of DirectShow filter. Insert it and connect its input pin to the decompressor and its output pin to the renderer. The sequence should appear again when you play the filter graph.

We will see latter how the ProxyTrans filter can be used to process the sequence. But since we want to transform the original sequence through some process, it might be useful to be able to save the processed sequence. Lets make some test using again the GraphEdit application. Delete the Video Renderer filter; we will replace it by a chain that will compress back the sequence and save it to a file. We therefore need a Video Compressor, an AVI multiplexor and a File Writer. You can easily find all these filters in the list of available filters when you click on the Insert Filter button. Note that when you select the File Writer filter, you will be ask to specify a name for the output file. The resulting graph should be as follows:

Obviously, if you play this graph, the resulting file will be the same as the original because our ProxyTrans filter that is supposed to do the processing does not do anything for now. However, the size of the output sequence might be different from the size of the original sequence, this is because of the compressor used in the graph that might use different parameters to compress the sequence. You probably also noted that when you play the graph, no sequence is displayed, simply because we removed the Renderer. It is quite easy to add an extra path to the filter in order to allow the simultaneous display and saving of the sequence. The Smart Tee is the filter you need. Add it and create the following graph:

Note that the Smart Tee filter has two output pins. The capture pin controls the sequence flow; the preview pin will receive frames only if extra computational resources are available. When processing a sequence, you could also use two Smart Tee filters, one to display the original sequence, the other to display the processed one; that is what we will do now when building manually our filter graph. As you can see in the figure above, the creation of a video processing filter graph requires connecting several filters together. Many lines have to be added to our createFilterGraph method; the probability of making an error becomes then quite high. However, a closer look at this method reveals that the same sequence is repeated several times, suggesting that some generic function could be introduced to help the programmer. Following this idea, we can write an addFilter utility function. This one will be called each time a new filter need to be created and connected to some filter of a graph. This function has the following signature:
    
bool addFilter(REFCLSID filterCLSID, 
               WCHAR* filtername, 
               IGraphBuilder *pGraph,
               IPin **outputPin, 
               int numberOfOutput);
The first parameter is the CLSID identifier that specifies which filter will be created. The second parameter is the name that will be given to this filter in the current graph. The third parameter is a pointer to the filter graph. The outputPin parameter is both an input and an output parameter. As an input, it contains a pointer to the output pin to which the filter to be created must be connected. When the function returns, this parameter will contain a pointer to the output pin(s) of the filter thus created; the number of output pins that needs to be created is given by the last parameter of this function. The function returns true if the filter has been successfully created and connected to the filter graph. This function can be written in a straightforward manner. First the filter is created using CoCreateInstance, then the input pin is obtained and is connected to the specified output pin. Once this done, the last step consists in obtaining the required number of output pins. The function is then as follows:
    
bool addFilter(REFCLSID filterCLSID, 
               WCHAR* filtername, 
               IGraphBuilder *pGraph,
               IPin **outputPin,  
               int numberOfOutput) {

  // Create the filter.
  IBaseFilter* baseFilter = NULL;
  char tmp[100];

  if(FAILED(CoCreateInstance(
                filterCLSID, NULL, CLSCTX_INPROC_SERVER,
                IID_IBaseFilter, 
                (void**)&baseFilter)) ||!baseFilter)
  {
    sprintf(tmp,"Unable to create %ls filter", filtername);
    ::MessageBox( NULL, tmp, "Error", 
                    MB_OK|MB_ICONINFORMATION );
    return 0;
  }

  // Obtain the input pin.

  IPin* inputPin= GetPin(baseFilter, PINDIR_INPUT);
  if (!inputPin) {

    sprintf(tmp,
       "Unable to obtain %ls input pin", filtername);
    ::MessageBox( NULL, tmp, "Error", 
                    MB_OK | MB_ICONINFORMATION );
    return 0;
  }

  // Connect the filter to the ouput pin.

  if(FAILED(pGraph->AddFilter( baseFilter, filtername)) ||
         FAILED(pGraph->Connect(*outputPin, inputPin)) )           
  {

    sprintf(tmp,
        "Unable to connect %ls filter", filtername);
    ::MessageBox( NULL, tmp, "Error", 
                    MB_OK | MB_ICONINFORMATION );
    return 0;
  }

  SAFE_RELEASE(inputPin);
  SAFE_RELEASE(*outputPin);

  // Obtain the output pin(s).

  for (int i=0; i<numberOfOutput; i++) {

     outputPin[i]= 0;
     outputPin[i]= GetPin(baseFilter, PINDIR_OUTPUT, i+1);

     if (!outputPin[i]) {

       sprintf(tmp,
         "Unable to obtain %s output pin (%d)", 
         filtername, i);
       ::MessageBox( NULL, tmp, "Error", 
                      MB_OK | MB_ICONINFORMATION );
       return 0;
     }
  }

  SAFE_RELEASE(baseFilter);

  return 1;
}
Using this function, it becomes easy to create a complex filter graph. The one we will build now will include the ProxyTrans filter (note that the header file initguid.h must be included to be able to use this filter). To be useful, this filter must do something. In fact, the objective of this filter is to give access to the programmer to each frame of the sequence that can thus be processed. This is realized through a callback function that is automatically called for each frame of the sequence. This callback function passes in argument a pointer to the current image, the user is then free to analyze and modify this image. Here is an example of a valid callback function that can be used with the ProxyTrans filter.
    
void process(void* img) {

  IplImage* image = reinterpret_cast<IplImage*>(img);

  cvErode( image, image, 0, 2 );
}
In order to have this function called, it must be registered to the ProxyTrans filter. This is simply done by calling this method of the IProxyTransform interface.
    
pProxyTrans->set_transform(process, 0);
Here is now the function that creates the filter graph that processes an input sequence and save the result in a file. Two preview windows are displayed, one for the original sequence, the other one for the out sequence.
    
bool createFilterGraph() {

  IPin* pSourceOut[2];
  pSourceOut[0]= pSourceOut[1]= NULL;

  // Video source
  addSource(ifilename, pGraph, pSourceOut);
      
  // Add the decoding filters
  addFilter(CLSID_AviSplitter, L"Splitter", 
                             pGraph, pSourceOut);
  addFilter(CLSID_AVIDec, L"Decoder", pGraph, pSourceOut);

  // Insert the first Smart Tee
  addFilter(CLSID_SmartTee, L"SmartTee(1)", 
                             pGraph, pSourceOut,2);
 
  // Add the ProxyTrans filter
  addFilter(CLSID_ProxyTransform, L"ProxyTrans", 
                             pGraph, pSourceOut);

  // Set the ProxyTrans callback     
  IBaseFilter* pProxyFilter = NULL;
  IProxyTransform* pProxyTrans = NULL;
  pGraph->FindFilterByName(L"ProxyTrans",&pProxyFilter);
  pProxyFilter->QueryInterface(IID_IProxyTransform, 
                              (void**)&pProxyTrans);
  pProxyTrans->set_transform(process, 0);
      SAFE_RELEASE(pProxyTrans);
      SAFE_RELEASE(pProxyFilter);

  // Render the original (decoded) sequence 
  // using 2nd SmartTee(1) output pin
  addRenderer(L"Renderer(1)", pGraph, pSourceOut+1);

  // Insert the second Smart Tee
  addFilter(CLSID_SmartTee, L"SmartTee(2)", 
                            pGraph, pSourceOut,2);

  // Encode the processed sequence
  addFilter(CLSID_AviDest, L"AVImux", pGraph, pSourceOut);
  addFileWriter(ofilename, pGraph, pSourceOut);

  // Render the transformed sequence 
  // using 2nd SmartTee(2) output pin
  addRenderer(L"Renderer(2)", pGraph, pSourceOut+1);
       
  return 1;
}

Check point #3: source code of the above example.

You will note that the output file produced by this program is quite big. This is simply because we are not using any compressor when the sequence is saved. This is because such filter can only be obtained through enumeration. This is discussed in the next section.

Top of the page

 

(c) Robert Laganiere 2011