RE: [linrad] Linrad

[Robert McGwier <rwmcgwier@xxxxxxxxxxxxxxx]

Leif:

Thank you very much for your insight and experience.
This should be a lot of fun to go through. I was
not careful enough when I wrote before (I apologize
for including all our remarks but it might make it
easier).  In my statement, in reality, "Markov"
should replace periodic.  The arrival time of the
pulse will be Markov rather than periodic.  That means
where it will arrive in time will be heavily
biased, but not completely determined.  For the
nearly periodic pulses, this will be a very big
gain over not using the information of the
history. That does not mean storing lots and lots
of samples, the history means keeping around a
few determined parameters.  The variance
of the arrival times will be one of the parameters
determined.  I will investigate the rain induced
static scenario as well as I had not considered
it.

I look forward to working this out and hopefully it
will  add something to this exercise.

Bob


-----Original Message-----
From: owner-linrad@xxxxxxxxxxxxxxxxxxxxxx
[mailto:owner-linrad@xxxxxxxxxxxxxxxxxxxxxx]On Behalf Of Leif Åsbrink
Sent: Friday, June 27, 2003 14:06
To: linrad@xxxxxxxxxxxxxxxxxxxxxx
Subject: RE: [linrad] Linrad


Bob,

> I am primarily interested in the short duration,
> nearly fixed rate pulse trains.
I am afraid you will find this is a mistake. A lot
of local qrn that can be removed with very high
precision is not nearly fixed rate.

Obviously static rain is completely random.

Powerline noise is generated by a spark gap (bad
insulator) that is connected via a capacitor
(the good insulators) to a high voltage 50/60Hz line.
Each time (100Hz repetition) the voltage goes
through zero and the derivative dU/dt is large,
there will be a series of pulses with a time
interval related to dU/dt, capacitances, ignition
and rest voltage of the sparc gap. There is some
uncertainty in the voltages that leads to a
randomization of the pulse intervals. Powerline
noise is pulse trains of maybe a dozen pulses. The
pulse trains repeat at 100/120Hz. There is no
precise timing.

Commutator motor noise, oil burner ignition arcs,
most of the qrn sources are not nearly fixed rate.

Farmers fences and car ignition noise are the qrn
sources with fixed repetition rate. I think one needs
a pulse removal algorithm that can handle random
distributed pulses well - the distance pulse to pulse
will seldomly be exact enough to allow phase information
from pulse to pulse anyway.

>  The longer duration ones the human ear does not
> like but seems to tolerate.
Maybe, but the longer ones should be removed as well:)


> As I have spent 37 years contesting
> and about 5 years going to rock concerts, I cannot
> tolerate this impulsive noise for long before I am
> completely fatigued due to the pain in my inner
> ear from the wideband spectrum of these pulses.
> So I am motivated!
OK. I think you will experience a generally lower noise
floor and more weak dx signals as a bonus:)

> Fortunately one of my areas of expertise is the
> automation of procedures such as the one you
> have brought to our attention.  What I will be
> attempting to do is build on the thing you have
> pointed out, and that is you can subtract a
> copy of the pulse as it is presented from
> the receiver, even though it has been stretched
> by the system, since you have found you can
> calculate or guess a few parameters and subtract
> the pulse sufficiently from the incoming signal.
Oooh! There is no guessing at all - the curve fitting
is very accurate indeed.

> To me this is a math problem.  I have a known signal,
> a short duration pulse, and it is my job to figure
> out three things:
>
> 1) it happened
> 2) what the channel has done to the pulse
The calibration procedure is there to set up a filter that
will make the channel do exactly what you want to the pulse.
This is done with very high accuracy and the pulse that
comes out will have 95 to 99% of its energy on a single
sample if it happens to be in phase with the sampling clock.
It is up to the user to select the pulse shape he wants to
have. This is a rather complex matter that involves dynamic
range (if 16-bit arithmetics is in use) sampling rate and
how anti aliasing is implemented.

It is the fact that the pulses are optimally shaped that
makes the simple procedure of Linrad reasonably efficient.


> 3) remodulate the pulse with the parameters
>    determined in 2)
> 4) subtract it.
>
> Number 1 and 2 are best done with determining
> the parameters of a joint probability distribution.
> Since these pulses tend to be nearly periodic,
> all one needs to know to tell a lot about where we are
> right now in the signal is what we have observed
> before.  Since these things can wiggle about and
> change with time, it is probabilistic in nature
> but this "history determines the present" is still
> good.  This kind of signal or "stochastic process"
> is a Markov process, so named after the Russian
> mathematician who described their properties.
> The process is a pulse has happened or not.
I will be extremely interested of the outcome of this
approach. For reasons given above I am afraid you will
run into difficulties but on the other hand you might be
able to remove periodic pulses better - and once that
is done it might be easier to get rid of the non-periodic
ones.

> The observation we get is through a (changing?)
> filter plus more gaussian like noise (thermal).  This
> means that we are not observing this Markov process
> directly but through a mask.  We can pick out
> some features (otherwise Linrad would not work!).
But Linrad does not have any "history determines the present"
built into it at all. Absolute nothing. Linrad knows exactly
what a pulse will look like. It does a curve fitting to extract
the parameters because although a pulse is precisely known, its
phase relative to the sampling clock is unknown as well as the
amplitude and polarisation. Once the parameters are found,
the pulse is removed completely. Complications arise when a notch
filter is inserted to get rid of some local station. Linrad does
not recalculate the pulse response accordingly, something that
should not be very time consuming because the notch filters
do not have to change rapidly. Linrad uses another strategy,
it renormalizes the pulses by taking the bandwidth reduction into
account. If 10% of the bandwidth is removed due to notches, the
pulse amplitude goes down by 10%. The weak oscillatory structures
around the pulse will cange very much more - but that does not
matter at all. The correct pulse is subtracted and the error in
mis-match of oscillatory behaviour will be on the frequencies
where the notches are. They add nothing at the frequency where
we look for weak signals:)

> This mask makes the system of pulses plus observations
> what is called a Hidden Markov Model.
>
> At the place I work, there was an algorithm discovered
> in the early 1960's.  It was rediscovered and popularized
> by a scientist named Dempster and is known as the
> Maximization/ Expectation algorithm.
>
> I believe this algorithm can be made to directly apply
> and that as a result, we can completely automate the
> determination of the channel and pulses jointly, and
> this will lead to less pulse energy getting passed the
> current process as the modeling will be dynamic.  This
> last piece is a "theorem":  That this algorithm will
> make the fewest mistakes in the minimize mean square error
> (power leakage passed the subtracter).
>
> I will be writing this software with Frank, AB2KT.
> We will write it C and will share it here.  Thanks
> a lot Leif for pointing out this to us!  Those few
> issues of QEX with Linrad and the SDR have changed
> my hamming quite a bit!

I hope we will be able to collect a library of
interesting qrm situations where different strategies
can be tested. Taking advantage of any currently unused
information (e.g. fixed repetition rate) will surely
allow an improved accuracy. The main source of difficulties
with the Linrad blanker is splatter and keying clicks from
strong stations. Wideband noise with quite different
characteristics. Fitting a model waveform that includes
splatter and keying clics to each strong signal will
allow a dramatic improvement on crowded HF bands. It is
beyond what I can do in 2 x 100 kHz bandwidth on a
Pentium III however;)

73

Leif  /  SM5BSZ





LINRADDARNIL