Professor Sampler’s Notes: The Shannon-Nyquist Sampling Theorem Part III

Topics Covered:

  • The Shannon-Nyquist Sampling Theorem
  • The Nyquist rate
  • A bandlimited signal vs a timelimited signal
  • Amplitude
  • The frequency spectrum and the bandwidth

Willingly would I burn to death like Phaeton, were this the price for reaching the sun and learning its shape, its size and its substance – Eudoxus of Cnidus

Fantastic! According to my earlier calculations, I concluded that if we matched the sampling frequency (the number of times we sampled per second) to the highest signal frequency (0.5 Hz) we could catch all the ‘events’ of Hertz’s walk. In other words, the sampling (one sample per second) is equal to Hertz’s walk (one step per second) Will it work? Let’s find out.

Instead of the whole path, let’s just take one section of it and test our theory. Here’s a section of Hertz’s walk (the green dots are sampling points):


The second image (grayed out) shows us what we will get if we sample from point ‘zero’. Aha! You are probably beginning to see a headache down the line. The third and last graph shows us what happens if we sample at the same frequency, but at ‘half-step’ instead of point ‘zero’. As you can see very clearly, this tiny shift makes a huge difference to what we get at the end. In both cases we are sampling at one step per second, yet we can get two results. Here’s the problem: we don’t always know when the crests, troughs and zero-crossings will take place. Not all waves are as well behaved as our postman’s!

Shocking, ain’t it? If a signal is sampled at the same frequency as its maximum frequency, one might miss the peaks and valleys completely!

Even if we shift the sampling time by a fraction so it matches with the peaks and valleys of the part with the maximum frequency, the part where frequency is lower doesn’t look too good. Is this the best we can do?

Since each step has two features – a rise and a fall, is it possible to double the rate of sampling when compared with the maximum frequency? Since Hertz is walking at 0.5 Hz, this means we could sample at 1 Hz (Two steps per second) and hope for the best. This was my last shot, as I’ve just about had enough of sampling. Here are my results:

What a difference! Whether I aligned the sampling points at the peaks or valleys or not, every frequency lower than the highest frequency was well represented, with the shape almost perfect! The highest frequency was perfect when the sampling aligned with its peaks and valleys, and was just short of the mark when not aligned. I was getting close!

What was I missing? I’ve been advising all along on how all these methods are subjective. We already know the universe never repeats itself – a perfect reproduction doesn’t exist. Therefore, the onus is on me to define just at what point my approximations are satisfactory. What did I want? I wanted to reproduce every important event in Hertz’s path. The first step towards this was to find out the highest frequency of his route, which I did.

The second step is to define what the events are. In this case, it’s simple, really. I have to know every zero crossing, crest and trough with accuracy. That’s the only way I can reproduce Hertz’s analog signal with accuracy enough to be proud of the exercise. That was the bar I set myself. Taking a close look at the last sampling exercise, I realized if I could fit in one more sample between each step, I could guarantee to have enough information to reproduce my wave. But I already know that if the samples are aligned very well, all I had to do was sample at twice the maximum frequency. And since sampling is a discrete activity – by which I mean you can’t half-sample, either you sample or you don’t – if I wanted to add another sample just to ensure that the wave is reproduced for those cases when the sampling is not in sync with the wave’s features at that point, I had to simply sample at more than twice the maximum frequency.

This is it. This is the Shannon-Nyquist sampling theorem, which simply states that a bandlimited analog signal can be perfectly reconstructed from an infinite sequence of equally spaced uniform samples if the sampling rate exceeds twice the highest frequency of the original signal. The key word here is ‘exceeds’. Just sampling at exactly twice the highest frequency isn’t good enough, as our example has shown. Never forget that.

I also imagined a scenario where Hertz would apply more pressure on some steps (when jumping or climbing down stairs) and lesser on others (when trying to outrun a dog). Here’s what such a signal would look like:


This difference in the height of each step is called the amplitude of the step, and considering the wave as a whole, the amplitude does not have to be constant, as is evident. After wavelength and frequency, amplitude is the third important part of a any wave. Tsunami waves have large amplitudes, ripples have small amplitudes.

When sampling at more than double the maximum frequency, these points were also well represented. If we sample right, we will have a discrete sampled signal from which we can hopefully recreate the original signal with an acceptable loss. This minimum sampling rate is called the Nyquist rate. According to the math, the analog signal can be recreated perfectly as long as the bandlimited signal is sampled at above the Nyquist rate. But practically it never works out that way.

The most important thing to note is that the Nyquist rate (greater than twice the maximum frequency of the analog signal) is the minimum condition. If one wants to represent an analog system ‘more perfectly’, one needs to sample it at frequencies higher than the Nyquist rate. The greater this frequency the better our chances of minimizing loss. That’s as good as it gets, but is a pain in the backside to achieve. Is this system perfect? Hell, no. And don’t ever kid yourself of this fact.

You might be wondering what the heck a bandlimited signal is. Suffice to say that a bandlimited signal is one whose Fourier transform (veggie-saur) has frequencies that can be constrained within a certain limit (hence the term bandlimit). But the sad fact is, this is just hoping for the best. We all know what happened at Isla Nublar, don’t we?

Our real-world signals are not bandlimited and can never be truly bandlimited, so what are they? Our T-Rex is what engineers call timelimited. Timelimited signals only exist for a short duration (short being either billions of years or nanoseconds or anything else!). A timelimited signal cannot at the same time be a bandlimited signal. This is the critical weakness of the sampling theorem. Waves in real life are timelimited (everything is, for that matter). The wave function is a function dependent on time (again, everything is!). The Fourier transform allows us to to convert this timelimited signal into a bandlimited signal (instead of time we are using frequencies – from T-Rex to Veggie-saur). This is where mathematics meets art, and the Fourier transform is to math what da Vinci’s work is to art. You better believe it.

Think about it this way: You can’t keep your T-Rex (timelimited wave function) and veggie-saur (Fourier transformed bandlimited function) in the same enclosure, any way you slice it. The math, of course, backs both sides somehow. Weird how it does that. But beware of mathematics, it is like a fair maiden who will bewitch you. The real world has a way of mocking all things perfect by showing how they’re not. This is what a bandlimited signal looks like:


It’s just another graph of all the frequencies in a signal represented by the letter B. The stretch from -B to +B is called the bandwidth of a signal. X(f) is the Fourier transform of the signal. See what the Fourier transform has done? It gives us a way to think of our signal in terms of frequencies, and the entire range of frequencies is called the frequency spectrum of the signal.

Wow. Just like we use terms like power, torque, mileage, etc. to define and get a handle on cars, we use frequency, amplitude, wavelength, wave function, frequency spectrum, etc. to define and get a handle on waves.

Don’t for an instant believe you have a complete grasp on the sampling theorem. This is just a basic primer into the highly complex world of sampling, and without a solid mathematical and engineering foundation one cannot hope to claim to be a master at it. Read the entries given in the links for further study, and you’ll begin to have a general idea of what this part of the world looks like.

I know we’ve covered a lot, but we still have the other half of our ‘deed’ to complete. All we have done till now is break up our signal into little chunks that we can confidently (to different degrees, let’s agree) say is enough to recreate our original signal. Our beloved T-Rex is now officially an endorsed Veggie-saur. Now I want him back. But how?

Takeaways:

  • The Shannon-Nyquist sampling theorem states that a bandlimited analog signal can be perfectly reconstructed from an infinite sequence of equally spaced uniform samples if the sampling rate exceeds twice the highest frequency of the original signal.
  • Amplitude is the magnitude of change in a wave. Really, depending on how you construct a wave, this magnitude can represent many things.
  • The frequency spectrum is a representation of a timelimited signal in the frequency domain. The frequency spectrum can be generated via a Fourier transform of the signal.
  • If the Fourier transform has finite support (no frequencies/energies beyond prescribed limits) then the signal is said to be bandlimited.
  • Any signal that can be represented as an amplitude that varies with time has a corresponding frequency spectrum.
  • Bandwidth is the difference between the upper and lower frequencies, and is usually measured in Hertz.

Links for further study:

Next: Professor Sampler’s Notes: Reconstruction
Previous: Professor Sampler’s Notes: The Shannon-Nyquist Sampling Theorem Part II