Shriphani Palakodety » DSP

More Updates to Listener

Shriphani — Mon, 15 Nov 2010 02:04:15 +0000

One of the rather irritating problems I was facing with Listener was that I had to set the intensity threshold for an alert manually (which is rather stupid). So I added a couple of lines that set the intensity threshold to what the average intensity of the surroundings currently is. So I no longer have to calibrate my app to fire alerts at the right intensity.

Besides this, Microsoft interview on Tuesday, thanksgiving break and finals approaching fast etc. etc. Whatever.

Listener Gets Another VAD

Shriphani — Mon, 09 Aug 2010 23:46:49 +0000

Over this summer, I wanted to make Listener Python only (there was a ton of needless applescript) and since my work @ VACCINE [VACCINE homepage] (more on this in a new post) didn’t leave much time for tinkering with things, I had to push this back as much as I could. I also added another voice activity detection algorithm which I believe works a lot better than what I had previously. And I also moved my project over to GITHUB.

The additions to Listener:

A new Voice Activity Detection algorithm [ link to pdf by Moattar and Homayounpour].
Skype4Py is still buggy on OS X with Python2.5 and it takes a nice solid dump (segfault) with Python2.6 so I am sure if the Skype part of the code works.
Python only. I removed the applescript portion from my code.

I have moved the entire code over to github: http://github.com/shriphani/Listener/ . The VAD algorithms can be seen in the file VAD.py.

This VAD also works by starting off with a base threshold for energy, power and an attribute called Spectral Flatness Measure. Luckily the paper had pseudocode so my DSP n00bishness wouldn’t get in the way of progress.

Anyway, to get this version of Listener running, download Listener.tar , untar it and run:

python2.5 audio_analysis.py

And you should be all set.

Listener Gets a VAD

Shriphani — Thu, 21 Jan 2010 01:15:23 +0000

So, the beginning of the 4th semester in the midst of losers and overachievers and this sem promises to set my a$$ on fire. As usual, I plan to continue working under Dr. Kihara this sem so that should be interesting. Anyway, I decided to improve upon what listener offered and decided to add a VAD algorithm to it. I initially chose the algorithm by moattar and homayounpur and decided that I ended up with too much to do (it might certainly be a good candidate for later, when I have more time for example). Hence, I decided to snoop around for something simpler and found this paper which seemed small and had a sort of ever changing threshold for successive frames. The paper was authored by S.Milanovic, Z. Lukac and A. Domazetovic. I still don’t think I got it exactly right though. The paper mentioned that they used counters to mark frames as silence based on what the previous frame was and I had to come with a counter upper bound for myself and finally chose to go with 10 as a good counter for such stuff. i.e. even if a particular frame doesn’t make it beyond its threshold, it still will be marked as active if the previous frame was active. This is done to accommodate situations where we end up reducing our volume at the end of a word / sentence.

Finally to decide whether there was speech on an overall level, I look for at least 3 instances of 18 consecutive frames being marked as active (just random picks, 18 frames allows 8 active frames and 10 additional for the counter we have and 3 looked like a good candidate at the end when I spoke my own name out).

And as a final measure, I also ensure that the overall intensity beats 48 dB so that someone trying to have a conversation with me is only recognized.

Finally, I made the switch from GeekTool to Growl as this thing kept taking a solid amount of real estate and since I have one 23” monitor and a 15” monitor, this thing is positioned outside the real estate of my laptop’s display. Growl seems like a better candidate overall and since I could get growl bindings to build on my machine finally, I think I should let growl handle this.

So, the only places where my VAD implementation (or my mod of whatever was in that paper) doesn’t seem to work is in surroundings with a piano (in our dorm’s lobby for example), v inconvenient but whatever, probably some time in the future, I will begin understanding DSP and spectral analysis well enough to come up with a simple VAD algorithm (as opposed to implementing something straight from a paper without any understanding of what is going on). Anyway, here is the updated script, it seems to do well recognizing speech in sort of silent settings:

#!/usr/bin/env python
#Author: Shriphani Palakodety
#Tool to aid those with noise cancellation headphones

import pyaudio
import wave
import sys
import struct
import numpy
import time

Growl_exists = True

try:
import Growl
except ImportError:
print "No Growl"
Growl_exists = False
pass

skype_on_call = False
notifier = 0
if Growl_exists:
notifier = Growl.GrowlNotifier(‘Listener’, [‘Attention’, ‘test’])
#notifier.applicationName = ‘Listener’
notifier.register()

def record():
”‘Records Input From Microphone Using PyAudio’”
duration = 3 #record for 1 second. Pretty long duration don’t you think
outfile = "analysis.wav"

p = pyaudio.PyAudio()

inStream = p.open(format=pyaudio.paInt16, channels=1, rate=44100,input=True, frames_per_buffer=1024)

out = []
upper_lim = 44100 / 1024 * duration #upper limit of the range we record to. 44100 / 1024 sized chunk * 5 seconds

for i in xrange(0, upper_lim):
data = inStream.read(1024)
out.append(data)

#now the writing section where we write to file
data = ”.join(out)
outFile = wave.open(outfile, "wb")
outFile.setnchannels(1)
outFile.setsampwidth(p.get_sample_size(pyaudio.paInt16))
outFile.setframerate(44100)
outFile.writeframes(data)
outFile.close()
analyze()

def analyze():
if skype_on_call:
print "\n"
print "Skype Call In Progress"
print "Listener On Hold"
return
inFile = wave.open("analysis.wav", "rb") #open a wav file in read mode
thresh = 1000 #establish a minimum threshold
max_samp = 0

decision = [0]

#for i in xrange(441):

inactive_counter = 0

vals = inFile.readframes(inFile.getnframes()) #read in 30 samples
len(vals)
results = struct.unpack("%dh"%(inFile.getnframes()), vals) #unpack to get the samples
results = [abs(x) for x in results]

#now we need to pull 30 samples at a time (30 samples = 1 frame).

for i in xrange(4404):
frame = results[30*i: 30*(i+1)]
print frame
new_thresh = (thresh * (1 – (2.0 ** -7))) + ((2 ** -8) * max_samp)

#check how many samples go above this new threshold
count = 0

for j in frame:
if j > new_thresh:
count += 1
if count / 30.0 >= 0.9 : #need it to beat 90%
#frame is a candidate for speech
decision.append(1)

else:
#this is where we use a counter based implementation for labelling inactiveness
if inactive_counter < 10 and decision[-1] == 1: #we ignore silence for 10 runs
decision.append(1)
inactive_counter += 1
else:
inactive_counter = 0
decision.append(0)

#update the threshold and the max sample values
thresh = new_thresh
max_samp = max(frame)

#final check for characterization as speech, we use another counter
active_counter = 0 #since the inactive counter will cause silence to be recognized as speech, we only consider speech as
print decision
final_num = 0
for val in decision:
if active_counter >= 18:
print "Speech!"
final_num += 1
active_counter = 0
if val == 1:
active_counter += 1
else:
active_counter = 0

results = [x ** 2 for x in results]
intensity = 20 * numpy.log10(numpy.sqrt(sum(results)/inFile.getnframes()))

if final_num >= 3 and intensity > 48:
if Growl_exists:
notifier.notify(‘Attention’,‘Listener’, ‘Speech Detected Nearby’)
else:
print "Speech Detected Nearby!\nSomeone might be calling you"
inFile.close()

if __name__ == "__main__":
f = open("skype_Status", "r")
for new_line in f:
if new_line == "PROGRESS":
skype_on_call = True

if skype_on_call:
analyze()
else:
record()

Anyway, it would be really convenient if I could find something about VAD algorithms and improve listener to work better for my dorm room settings. It is doing a pretty good job already but there is always scope for improvement.

As always, my solutions need to be convoluted and over here, I make use of applescript to check if there’s a skype call going on or not, so yeah, you can find all that here.

Screenshots etc available on Listener’s new home: http://shriphani.com/blog/listener/.

First DSP Attempts

Shriphani — Fri, 26 Jun 2009 13:51:02 +0000

Well, since my last post on detecting calls of attention using my microphone, I have been paying attention to DSP since it seems to be one cool topic to spend 3 months on. So, I decided to begin reading the dspguide and found some cool stuff which I decided to tinker with. After reading about convolution, I decided to implement some basic filters that would help me amplify, add echoes and so on. First, implementations of convolution, the input side algorithm and the output side algorithm:

The input side algorithm:

def convolute_inside(impulse_response):
global output_signal
global input_signal
len_out_signal = len(input_signal)+len(impulse_response)-1
output_signal = [0 for x in xrange(len_out_signal)]
for i in xrange(len(input_signal)):
for j in xrange(len(impulse_response)):
output_signal[i+j] = output_signal[i+j] + input_signal[i]*impulse_response[j]

The output side algorithm:

def convolute_outside(impulse_response):
global output_signal
global input_signal
len_out_signal = len(input_signal)+len(impulse_response)-1
for i in xrange(len_out_signal):
output_signal.append(0)
for i in xrange(len_out_signal):
for j in xrange(len(impulse_response)):
#print i, j
if not (i-j)<0 and not (i-j)>len(input_signal)-1:
output_signal[i] += impulse_response[j] * input_signal[i-j]

Next, using appropriate filters, we can add echoes and amplify the input signal. So, we first need to read in a wav file. Using my previous script, that should be simple:
Reading a wav file:

def readWavFile(input):
inFile = wave.open(input, "rb")
sample_rate = inFile.getframerate()
total_samples = inFile.getnframes()
vals = inFile.readframes(total_samples)
inFile.close()
return struct.unpack("%dh"%(total_samples), vals)

This next function adds an echo to the input signal:
Add an echo:

def add_echo():
”‘Amounts to scaling and shifting the delta function and then adding it to a delta function’”

#considering the intensity of the echo to be 60% of that of the original signal and delayed by 1000 samples:

impulse_response = [0 for x in xrange(1003)] #shifted and scaled delta function + delta function
impulse_response[0] = 1
impulse_response[-1] = 0.6
convolute_inside(impulse_response)

The above function considers that an echo occurs a 1000 samples after the current one and with an intensity 6 tenths of the original signal. Since testing this becomes a serious pain, I decided to test my echo function using a different input signal and impulse response (echo is 6/10 of the original intensity and 4 samples away).

def add_new_echo():
”‘Amounts to scaling and shifting the delta function and then adding it to a delta function’”

#considering the intensity of the echo to be 60% of that of the original signal and delayed by 4 samples:

impulse_response = [1,0,0,0,0.6] #shifted and scaled delta function + delta function
convolute_inside(impulse_response)

input_signal = [1,2,3,4]
#print input_signal
add_new_echo()
print output_signal

The output of this is:

mouse-den% python convolution.py
[1, 2, 3, 4, 0.59999999999999998, 1.2, 1.7999999999999998, 2.3999999999999999]

The impulse response for amplifying a signal would be a scaled delta function, so:

def amplify():
”‘Make the impulse function a scaled delta function’”

impulse_response = [2]
convolute_inside(impulse_response)

The code to write to a wave file should be pretty straightforward:

def writeWavFile(out):
outStream = wave.open(out, "wb")
outStream.setnchannels(1)
outStream.setsampwidth(2)
outStream.setframerate(44100)
data = ""
for i in xrange(len(output_signal)):
data += struct.pack(‘h’, output_signal[i])
outStream.writeframes(data)
outStream.close()

So, here is the test wave file: test.wav

And the amplified wav file: output1.wav

Now, to the part where we try to figure out if a signal contains another signal.

Turns out, to do that, you just need to obtain the cross-correlation of the input signal and the waveform we already have.

So, the code to do this for a signal that goes like [1,2,3,4] is:

def cross_correlate(input_file=""):
”‘Cross correlate, obtain the graph and find the peak’”

global input_signal
global output_signal
#input_signal = record() #this procedure reads in the signal we need to find
input_signal = [1,2,3,4,11,0,0,1,10,5,2]
impulse_response = [1,2,3,4] #read in our signal
impulse_response.reverse()
convolute_outside(impulse_response)
print output_signal

This results in a signal that looks like:

[4, 11, 20, 30, 64, 44, 26, 15, 43, 52, 44, 26, 9, 2]

So, I had to figure out an algorithm to detect the peak and I found one on stackoverflow.com and it goes like this:

for i in xrange(len(output_signal)-1):
travel = output_signal[i+1] – output_signal[i]
rise = output_signal[len(output_signal)-1] – output_signal[0]

if (travel/rise) > 1:
#print output_signal[i]
peak_options[output_signal[i]]=i

return peak_options[max(peak_options.keys())]

In other news, I am going to work in Prof. Kihara’s bio-informatics laboratory this fall and I hope it works out.