c++, Kinect

Kinect V2 – Simple Sound Detection C++

This is a very simple post showing an implementation of the detection of a sound, any sound above a threshold, using the Kinect V2 SDK. The sample is written as a C++ XAML Windows Store app. The code uses the usual pattern for retrieving data from the Kinect via the SDK; that is,

Get Sensor

Choose Datasource

Open a Reader on the DataSource

Subscribe to the FrameArrived event on the Reader

Open the Sensor

This looks like this:

  1. void MainPage::OnLoaded(Platform::Object ^sender, Windows::UI::Xaml::RoutedEventArgs ^e)
  2. {
  3.     _kinect = KinectSensor::GetDefault();
  4.  
  5.     auto audioSource = _kinect->AudioSource;
  6.  
  7.     // Allocate byte array to hold the audio data
  8.     _audioBuffer = ref new Array<byte>(audioSource->SubFrameLengthInBytes);
  9.  
  10.     _reader = audioSource->OpenReader();
  11.  
  12.     _reader->FrameArrived += ref new Windows::Foundation::TypedEventHandler<WindowsPreview::Kinect::AudioBeamFrameReader ^, WindowsPreview::Kinect::AudioBeamFrameArrivedEventArgs ^>(this, &MainPage::OnFrameArrived);
  13.     _kinect->Open();
  14. }

 

The the FrameArrived handler goes something like this:

Get BeamFrame Collection

GetBeamFrame Collection first member and iterate over the SubFrames within

For each valid SubFrame copy out the Audio data

Process the audio data

  1. void MainPage::OnFrameArrived(WindowsPreview::Kinect::AudioBeamFrameReader ^sender, WindowsPreview::Kinect::AudioBeamFrameArrivedEventArgs ^args)
  2. {
  3.     auto beamCollection = args->FrameReference->AcquireBeamFrames();
  4.     if (beamCollection == nullptr)
  5.         return;
  6.     if (beamCollection->Size < 1)
  7.         return;
  8.  
  9.     for (auto subFrame : beamCollection->GetAt(0)->SubFrames)
  10.     {
  11.         if (subFrame == nullptr)
  12.             continue;
  13.  
  14.         // Continue until timeout exceeded
  15.         if (_lastDetected.Duration != 0.0 && (subFrame->RelativeTime.Duration < _lastDetected.Duration + TIMEOUT))
  16.             continue;
  17.  
  18.         subFrame->CopyFrameDataToArray(_audioBuffer);
  19.  
  20.         double sum = 0;
  21.         int stride = sizeof(float);
  22.         float sq = 0.0f;
  23.         int sampleCounter = 0;
  24.  
  25.         // Loop over audio data bytes…
  26.         for (int i = 0; i < _audioBuffer->Length; i += stride)
  27.         {
  28.             memcpy(&sq, &_audioBuffer[i], sizeof(float));
  29.             sum += sq * sq;
  30.             sampleCounter++;
  31.             if (sampleCounter < 40)
  32.                 continue;
  33.             float energy = (float) (10.0 * log10(sum / sampleCounter));
  34.             sampleCounter = sum = 0;
  35.  
  36.             if (energy > ENERGY_THRESHOLD)
  37.             {
  38.                 _lastDetected = subFrame->RelativeTime;
  39.  
  40.                 // Sound detected!! Do Stuff….
  41.                 break;
  42.             }
  43.         }
  44.     }
  45. }

The sample is here https://github.com/peted70/kinectv2-detectsound.git

Tagged , ,

Leave a Reply

Your email address will not be published. Required fields are marked *