Introduction
Musical instrument
signals generally consist of a transient portion and steady state or quasi-periodic
portion. The transient part is usually the attack of the signal and the
steady state the portion that follows the attack part. When investigating
time variant signals it is critical to make use of both time and
frequency domain analysis techniques. Some important features in musical
signals include duration, amplitude modulation, pitch, spectral harmonicity,
spectral envelope, spectral centroid and the like. Attack time is especially
considered a salient feature of musical timbre (Eagleson and Eagleson 1947;
Saldanha and Corso 1964; Elliot 1975) and has been thought to be a dominant
feature of musical instruments. However, it has also been discovered that
the attack time and also note-to-note transients of a signal are neither
sufficient nor necessary for recognizing
musical instruments (Kendall 1986). This controversial discovery supports
the importance of the steady state portion of a signal.
signals generally consist of a transient portion and steady state or quasi-periodic
portion. The transient part is usually the attack of the signal and the
steady state the portion that follows the attack part. When investigating
time variant signals it is critical to make use of both time and
frequency domain analysis techniques. Some important features in musical
signals include duration, amplitude modulation, pitch, spectral harmonicity,
spectral envelope, spectral centroid and the like. Attack time is especially
considered a salient feature of musical timbre (Eagleson and Eagleson 1947;
Saldanha and Corso 1964; Elliot 1975) and has been thought to be a dominant
feature of musical instruments. However, it has also been discovered that
the attack time and also note-to-note transients of a signal are neither
sufficient nor necessary for recognizing
musical instruments (Kendall 1986). This controversial discovery supports
the importance of the steady state portion of a signal.
This chapter
mainly describes the implementation of the signal processing algorithms
used in the software system for extracting features that depict these transient
and stationary characteristics in the frequency and time domain. The frequency
domain analysis section of this chapter is primarily based on the discrete
Fourier transform (DFT). DFT based spectral analysis algorithms discussed
includes
short time Fourier transform, spectral centroid, spectral
smoothness andtracking
of partials over time. In the time domain analysis section I will
mainly describe the implementation of algorithms including pitch detection
with interpolation and a period averaging based on the autocorrelation
function. Other modules discussed are amplitude envelope, amplitude modulation,
attack time computation and noise content analysis.
Frequency
Domain Analysis
Domain Analysis
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">DFT<br /> and STFT</font>
</div>
</ol>
The spectral
analysis part of feature extraction is primarily based on the discrete
Fourier transform (DFT). Below the continuous time and discrete time versions
of the Fourier transform are shown.
analysis part of feature extraction is primarily based on the discrete
Fourier transform (DFT). Below the continuous time and discrete time versions
of the Fourier transform are shown.
(2.1)
(2.2)
To extract
transitory spectral characteristics the short time Fourier transform (STFT)
was used (Allen 1977; Allen and Rabiner 1977). The basic algorithm is as
follows.
<p>
<span style=""><img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49ha.gif" name="Object5" width="428" height="44" hspace="8" /><font face="Arial, sans-serif">(2.3)</font></span>
</p>
<p>
</center>
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">As seen in<br /> figure 2.1 the STFT can be simply described as windowing and taking the<br /> FFT of the signal. There are various window types available in the program</font>
</p>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hb.gif" name="Graphic10" width="552" align="left" border="0" height="107" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.1 Short time Fourier transform<br /> and Spectral Peak Detection</font>
</center>
</td>
</tr>
</table>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">with different<br /> side-lobe and main lobe characteristics.<span style="font-style: normal;"><br /> The Hamming window has been shown to work particularly well with<br /> <i>musical</i><br /> signals (De Poli, Piccialli and Roads 1991). See the appendix for details<br /> regarding windowing and its side-lobe and main lobe characteristics.</span></font><br /> <br /> <br /> <br /> <br /> <br />
</p>
<ol start="2">
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Spectral<br /> Peak Detection and Tracking</font>
</div>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Pitched musical<br /> instruments display a high degree of harmonic spectral quality when analyzed<br /> for frequency content. Most tend to have<br /> <i>quasi-integer</i> harmonic<br /> relationships between spectral peaks and the fundamental frequency. In<br /> voice, the spectral envelope displays mountain-like contours or valleys<br /> known as formants. The locations of the formants distinctively describe<br /> vowels. This is also evident in violins, but the number of valleys is greater<br /> and the formant locations change very little with time unlike the voice,<br /> which varies substantially for each vowel. Woodwinds such as the bassoon<br /> and oboe on the other hand have fewer formants than the voice, but tend<br /> to have stronger and clearer spectral contours that perceptually characterize<br /> the woodwind family (Cook 1999). Generally, musical instruments like the<br /> plucked string (figure 2.2) exhibit lower energy in the high frequency<br /> bins. The higher partials normally have less energy and also die out faster<br /> than lower ones over time.</font>
</div>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hc.gif" name="Graphic19" width="521" align="left" border="0" height="420" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<span style="font-style: normal;"><font face="Arial, sans-serif">Figure<br /> 2.2 Plucked string spectrum</font></span>
</center>
</td>
</tr>
</table>
<p>
<font face="Arial, sans-serif">Using the short time Fourier transform,<br /> I have implemented a spectral peak detection and tracking method, extracting<br /> quasi-integer related harmonics from the spectrum. The peak picking algorithm<br /> takes into consideration magnitude and frequency information to select<br /> the most prominent and harmonically behaving peaks. To help in the search<br /> for spectral peaks, various threshold values are used as described below.</font><br /> <br /> <br /> <br /> <br /> <br />
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">The spectral<br /> peak detection algorithm is divided into four main steps. The first pass<br /> roughly locates possible peaks, where the roughness factor for searching<br /> peaks is controlled via a <span style="font-style: normal;">threshold<br /> value</span>. The threshold value basically dictates the degree of “peakiness”<br /> that is allowed for a local maximum to be considered a possible peak. The<br /> second pass filters out peaks that may have been erroneously selected in<br /> step<br /> 1. The third pass looks for any <i>broken harmonic sequence,</i> analyzing<br /> harmonic relationships of the currently selected peaks. In this pass, peaks<br /> that may have been deleted or missed in the previous two passes are inserted.<br /> The final pass looks at the selected peaks and further does a harmonic<br /> analysis ultimately leaving a set of peaks that are most <span style="font-style: normal;">probably</span><i><br /> harmonics. </i>A <i>mean</i> and <i>scalable standard deviation error</i><br /> method is applied for control of inharmonicity.</font><br /> <br />
</p>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hd.gif" name="Graphic23" width="576" align="left" border="0" height="42" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.3 Peak detection algorithm</font>
</center>
</td>
</tr>
</table>
<ol start="2">
<ol>
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Step<br /> 1: Rough Peak Detection</font>
</div>
</ol>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<span style="font-style: normal;"><font face="Arial, sans-serif">In<br /> the rough peak detection algorithm p</font></span><font face="Arial, sans-serif">ossible peaks are picked using<br /> negative and positive slope threshold values to guide in the selection<br /> process. As shown in figure 2.4<span style="font-style: normal;"> the polarity<br /> of the slope of the spectrum is computed from bin to bin (DC to Nyquist)<br /> using the basic assumption that a transition from positive to </span></font>
</div>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49he.gif" width="486" border="0" height="385" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.4 Rough search for peaks</font>
</center>
</td>
</tr>
</table>
<div style="font-style: normal; line-height: 200%;">
<br /> <br /><font face="Arial, sans-serif">negative slope calls for the possibility<br /> of a peak. The following conditions help in the selection of a peak:</font>
</div>
<p>
<br /> <br /> <br /> <br />
</p>
<ol>
<div style="font-style: normal; line-height: 200%;">
<font face="Arial, sans-serif">The<br /> slope must change polarity, positive to negative.</font>
</div>
<div style="font-style: normal; line-height: 200%;">
<font face="Arial, sans-serif">The<br /> magnitude difference between the peak candidate and the <i>current</i><br /> bin’s magnitude component (<i>X[k]-X[k+4]</i>) must be greater than a threshold<br /> value – see example (figure 2.5).</font>
</div>
<div style="font-style: normal; line-height: 200%;">
<font face="Arial, sans-serif">A<br /> new peak candidate search occurs only after there is a slope change from<br /> negative to positive and when a threshold value as shown in figure 2.6<br /> is exceeded.</font>
</div>
</ol>
<div style="font-style: normal; line-height: 200%;">
<font face="Arial, sans-serif">Refer<br /> to flowcharts in the appendix for details.</font>
</div>
<p>
</p>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hf.gif" width="318" border="0" height="340" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.5 Actual peak assessment</font>
</center>
</td>
</tr>
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hg.gif" width="487" border="0" height="328" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.6 Transitional peaks (noise)</font>
</center>
</td>
</tr>
</table>
<p>
</p>
<ol start="2">
<ol>
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Step<br /> 2: Prominent Peak Search</font>
</div>
</ol>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">In step 2,<br /> <i>prominent<br /> peaks</i> are located from a set of potential peaks found in step 1. The<br /> purpose is to filter out local peaks which may be present<br /> <i>between</i><br /> stronger partial candidates as shown in figure 2.7. The search for <i>prominent<br /> peaks</i> is done in the following way:</font>
</div>
<p>
</p>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hh.gif" width="546" border="0" height="425" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.7 Prominent peak search</font>
</center>
</td>
</tr>
</table>
<ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">The bin with<br /> the maximum magnitude is found.</font>
</div>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Relative<br /> to position of the peak with maximum amplitude, peaks are analyzed moving<br /> towards DC.</font>
</div>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Relative<br /> to position of peak with maximum amplitude, peaks are analyzed moving towards<br /> the Nyquist frequency.</font>
</div>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Local maxima<br /> or peaks are picked out using an adaptive threshold value that is reflective<br /> of a prominent<i> peaks </i>(possible<br /> partials) and its neighboring peaks as shown in figure 2.7. For<br /> example a 50% threshold value will require neighboring peaks to be greater<br /> than at least half the magnitude of the <i>prominent peak</i> (possible<br /> partial). Refer to the appendix for details on algorithm.</font>
</div>
<p>
<br /> <br /> <br /> <br />
</p>
<ol start="2">
<ol>
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Step<br /> 3: Harmonic Break Search</font>
</div>
</ol>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">The third<br /> step is called the <i>harmonic break</i> search. Here, I have tried to<br /> analyze if some “potential partials” were deleted or missed in the previous<br /> steps. This may occur when potentially harmonically related peaks temporarily<br /> have little energy or are simply much weaker than the stronger ones, but<br /> are nevertheless harmonic. The <i>harmonic break search</i> is divided<br /> into the following sub-routines:</font>
</div>
<ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Analyze harmonic<br /> relationship between current partial candidates, by computing the mean<br /> bin spacing between all prominent peaks.</font>
</div>
<p>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hj.gif" name="Object22" width="196" height="44" hspace="8" /><font face="Arial, sans-serif">(2.4)</font>
</center>
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">Detecting any<br /> <i>harmonic<br /> breaks</i>, or discontinuities between prominent peaks.</font>
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">If discontinuities<br /> are found, going back to step 1 and 2 and do a refined search of possible<br /> peaks between pairs of <i>prominent peaks</i>.</font>
</p>
</ol>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hk.gif" width="565" border="0" height="356" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.8 Harmonic break search</font>
</center>
</td>
</tr>
</table>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">In the<br /> <i>harmonic<br /> break search’s</i> second step, harmonic discontinuities are detected using<br /> a pair of threshold values limiting the range of harmonic deviation. Hence,<br /> the algorithm expects the possibility of a peak within the threshold bounds<br /> computed in sub-step 2 (figure 2.8). Refer to appendix for more details<br /> on algorithm.</font><br /> <br /> <br /> <br /> <br /> <br />
</p>
<ol start="2">
<ol>
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Step<br /> 4: Harmonicity Analysis</font>
</div>
</ol>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Finally in<br /> step 4 an overall harmonicity verification is performed. In this last step,<br /> the first few peaks (selectable in software) are used as a guide to determine<br /> the final set of <i>partials</i>. The reason for choosing the first few<br /> peaks of the spectrum is due to the fact that in highly pitch salient signals,<br /> lower harmonics usually are stronger and more stable than higher ones.</font>
</div>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">The idea<br /> is to use the gaussian normal distribution function employing mean, variance<br /> and standard deviation for eliminating inharmonic or misbehaving partials.<br /> A peak that is outside a right and left threshold bound is considered inharmonic<br /> and misbehaving. A mean bin spacing value denoting the bin distances between<br /> neighboring peak candidates is computed to render the <i>variance</i> and<br /> <i>standard<br /> deviation</i>. As the lower partials generally tend to be more stable and<br /> have more energy, the first K (K: integer > 0) peaks are used for the computation<br /> of the standard deviation. A scaled version of the the standard deviation<br /> is then used as a criterion for evaluating inharmonicity of each partial<br /> candidate. The <i>scaledstandard deviation </i>is<br /> increased or decreased to control the permitted spread of each peak.<br /> In other words, the <i>scaled</i> <i>standard deviation</i> is directly<br /> relevant to the amount of inharmonicty <span style="font-style: normal;">tolerated</span><br /> for selecting the final set of peaks. The scalar that controls the<br /> <i>scaled<br /> standard deviation</i> is a value between 0 and 1, where 1 is equivalent<br /> to limiting the peaks to the original <span style="text-decoration: none;">un-scaled</span><span style="font-style: normal;"><br /> standard deviation.</span> This method is implemented by computing an ideal<br /> sequence of harmonics using the above acquired data. Hence the ideal harmonic<br /> series is a sequence of partials as shown below.</font>
</div>
<p>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49hm.gif" name="Object25" width="385" height="61" hspace="8" /><font face="Arial, sans-serif">(2.5)</font>
</center>
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">The ideal set<br /> of harmonics and the actual set of harmonics are compared and the error <img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49ho.gif" name="Object29" width="15" align="abscenter" height="20" hspace="8" />(equation<br /> 2.6) for each peak is computed and verified against the <i>scaled standard<br /> deviation</i> for final assessment. Peaks that have excessive error values<br /> are deleted from the final set of peaks and the remaining ones are finally<br /> considered <i>harmonics</i>. See the appendix for more details on algorithm.</font>
</p>
<p>
<center>
</p>
<p>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49i0.gif" name="Object24" width="334" align="abscenter" height="21" hspace="8" /><font face="Arial, sans-serif">(2.6)</font>
</p>
<p>
</center>
</p>
<p style="line-height: 200%;">
<font face="Arial, sans-serif">Equation 2.6<br /> shows the error between the ideal and actual bins where M is the number<br /> of ideal peaks and N is the number of actual peaks in the spectrum. M and<br /> N have different values as missing partials may exist in the actual set<br /> of peaks.</font><br /> <br /> <br /> <br /> <br /> <br />
</p>
<ol start="2">
<ol>
<ol>
<ol>
<div style="line-height: 200%; text-decoration: none;">
<font face="Arial, sans-serif">Partial<br /> Tracking between Frames</font>
</div>
</ol>
</ol>
</ol>
</ol>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">Once harmonics<br /> have been evaluated in each frame (a frame is equal to the length of the<br /> FFT), they are combined to render a spectrogram. Frame to frame partial<br /> movement is determined using a <i>harmonic continuity</i> criterion as<br /> shown in figure 2.9.</font>
</div>
<div style="line-height: 200%;">
</p>
<table width="100%" border="1" cols="1">
<tr>
<td>
<center>
<img loading="lazy" decoding="async" src="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/sv9h49i1.gif" width="486" border="0" height="286" />
</center>
</td>
</tr>
<tr>
<td>
<center>
<font face="Arial, sans-serif">Figure 2.9 Partial tracking between<br /> frames</font>
</center>
</td>
</tr>
</table>
</div>
<div style="line-height: 200%;">
<font face="Arial, sans-serif">The harmonic<br /> continuity criterion is explained as follows: Each harmonic in a frame<br /> is allowed to sway in frequency within a set of error margin values. Hence,<br /> as shown in figure 2.9, four of the harmonics make a continuous <i>harmonic<br /> path </i>(k, k+1, k+2, k+3).<br /> However, the harmonic in frame k+4 exceeds the allowed error margin and<br /> breaks the previous harmonic path. At frame k+4 a new path is created and<br /> the path which started at frame k is discontinued. The harmonic continuity<br /> criterion is helpful in observing movements of the harmonics over time<br /> and frequency.</font>
</div>
<p>
출처: <a href="http://silvertone.princeton.edu/%7Epark/thesis/dartmouth/html/ch2-1.html" target="_blank">http://silvertone.princeton.edu/~park/thesis/dartmouth/html/ch2-1.html</a>
</p>