A real example of e-mail over HF (via FTP) using STANAG-5066 and MS188-110A

in this recording two stations run STANAG-5066 with BFTP protocol client (a File Transfer Protocol) in order to exchange e-mails using the MS188-110A serial waveform. Note that the E-mail delivery does not happen here using STANAG-5066 HMTP client but rather by FTP_ing the email file (containing all the SMTP headers and fields) ready to be processed by common e-mail clients such as Microsoft Outlook. 
It is worth mentioning that BFTP (Basic File Transfer Protocol) client is defined by  STANAG-5066 Annex F.10.2.2, that same Annex also defines the Compressed File Transfer Protocol client (CFTP) that provides the most bandwidth-efficient exchange of data just using BFTP to send compressed file (see the comment about "MIL-STD-188-141B Notice 1" below at the end of the post).
Talk about e-mail and file-transport protocols go beyond the purpose of the blog so I prefer to have an  HF approach, keeping in mind that STANAG-5066 is designed for running IP applications over HF (STANAG-5066 IP based networks can be thought of as being an HF radio based version of the Internet) and that the stuff, in this sample, is arranged as in pic. 1.

pic.1

As said above, the heard waveform is a standard MS-188-110A serial, as can be verified by SA (pic. 2) although a little shift of the sub-carrier from the nominal 1800 Hz. Since at this stage the signal is coming directly from the USB demodulator, we face Over The Air (OTA) symbols. The structure of the MS188-110 frame is recognizable from the bitstream returned by the SA phase-plane demodulator after its conversion (pic. 3).

pic.2

pic.3 - OTA bitstream after demodulation performed by SA

In order to dig the signal we need de-scramble and de-interleave it and then  remove the extra bits added by the FEC encoder: a basic decoder will do the job returning the bitstream after the MS188-110 removal (pic. 4). 

pic.4 - the bitstream after the MS188-110 removal

Once detected the presence of STANAG-5066 as "secondary" protocol, we need to remove also its encapsulation so to get  the email message that have been transferred by BFTP protocol at sender side. Note that the name of FTP protocol is HBFTP, as it appears in the output window of the analyzer (pic.5). 

pic.5 - the bits trasferred by HBFTP after removed STANAG-5066 and MS188-110A

The acronym HBFTP could stand for HF-BFTP and is suggested to be the Harris implementation of this protocol, available in mail-gateway as the HARRIS RF-6760W Wireless Message Terminal or RF-6750W Wireless Gateway.
The contents of 000_HBFTP--3  can be saved to a file and Windows assigns the .eml extesion since the file exhibits all the features as it was received from a conventional SMTP server via an Internet connection (pic.6).  The file can be  opened and processed by Microsoft Outlook w/out problems: Outlook simply does not care where and how this file has arrived (pic. 7). For reasons of confidentiality the email addresses have been deliberately blackened.

pic.6

pic.7

MIL-STD-188-141B (change notice 1, Appendix E) defines a version of email specially adapted to HF communications. Commands to and from the server are aggregated into blocks to overcome the high latency introduced by HF transmission methods. This greatly improves the efficiency of email when carried over HF.
[...]

E.5.2.1 Compressed file transfer protocol.

The Compressed File Transfer Protocol (CFTP) sends compressed e-mail over an HF link using a file transfer protocol, rather than a mail transfer protocol. Messages produced by an email application are processed by a MTA, compressed in CFTP, segmented in the STANAG 5066 Basic File Transfer Protocol (BFTP), and passed to the subnet interface by the STANAG 5066 Reliable Connection Oriented Protocol (RCOP). At the receiving node, this process is reversed, and the uncompressed e-mail message is delivered to the receiving MTA for delivery or forwarding.
[...]

At this purpose, it is worth noting that the name of the compressed file (.gz extension) produced by CFTP is visible in the non-sense output of the MS188-110A decoder  shown in pic. 8

pic.8

The document "MIL-STD-188-141B Notice 1" can be downloaded from everyspec.com
 

Ital-HDLC/QUEDRE (High Level Data Link Control)

Looking at the Data Link layer, the many and seemingly boring STANAG-4285 transmissions start to get interesting: the following is an example about the pair S-4285 and HDLC.
The HDLC (High Level Data Link Control) is a group of protocols for transmitting synchronous data packets between Point-to-Point nodes and operates at the data link layer of the OSI reference model. The protocol uses the services of a physical layer (for example STANAG-4285 or MS188-110 waveforms) , and provides either a best effort or reliable communications between the transmitter and receiver (i.e. with acknowledged data transfer as ARQ). The type of service provided depends upon the HDLC mode which is used.

Each piece of data is encapsulated in an HDLC frame (pic. 1) by adding a trailer and a header. The header contains an HDLC address and an HDLC control field. The trailer is found at the end of the frame, and contains a Cyclic Redundancy Check (CRC) which detects any errors which may occur during transmission. The frames are separated by HDLC flag sequences which are transmitted between each frame and whenever there is no data to be transmitted (idle phase).
There is much documentation about it in the web, some links are given at the bottom.

pic. 1 - HDLC Frame Structure showing flags, header 
(address and control), data and trailer (CRC-16)

Sample HDLC data signal follows, when not sending data, a hardware generated idle pattern is present on the data signal:

[... idle pattern ...][Flag][...Data...][CRC][Flag][... idle pattern...]

As said, HDLC can be met in STANAG-4285 and MS188-110 as "secondary" (transported) protocol: the following is an example of its detection just in a STANAG-4285 transmission (pic. 2).

pic. 2 - a common STANAG-4285 transmission as seen by SA

Since I'm investigating the Data Link, I need a bitstream after having de-interleaved and removed the overhead bits added by the underlaying waveform; in other words, I need a STANAG-4285 decoded output. After identifying the correct settings for data-rate and interleaver, 1200bps/short in this case, I ran a k500 session and got the bitstream coming from the upper layer (pic. 3) which I then saved in a ASCII-bits file.

pic. 3 - bistream of that same transmisssion after stanag-4285 removal

The HDLC flags are easily identified looking for the "01111110" sequences (pic. 4) and the "presence" of this protocol has been confirmed (pic.5):

pic. 4

pic. 5

Things work in the same "logical" way TCP/IP stack does: as the data bits (the payload) flows down along a layer (transmitting-phase), they are encapsulated by (one of) the protocol running at that layer and this packet, data and protocol overhead, just forms the payload for the protocol that works at the next  layer. In this sample it's something like:

In the receiving-phase the packets go up the stack and are progressively de-encapsulated in order to return the starting payload, ie.e the original user-data.
That said, after removing Stanag-4285 and then the HDLC stuff, I got the sent messages (pic. 6).

pic. 6 - HDLC messages

Frames are recognized as idle sequences (idle lin) and are characterized by the repetitions of the string "QUEDRE" closed by "XT". Other than the frame type, the bitstream analyzer correctly detects the HDLC blocks, sizes and computes CRC (pic. 7).

pic. 7

The word "QUEDRE" does not belong to the embedded commands of HDLC, rather it seems a sort of an agreed string used for the idle sequences (at least here). That string is anyway one of the distinctive signs, since some bit analyzers tools use indifferently both the terms Ital-QUEDRE and Ital-HDLC just to indicate this feature in respect of the standard HDLC protocol. Difficult to say what QUEDRE means, perhaps a sort of acronym or most likely an agreed term, as said above, but in lack of a safe source these are only speculations. Since there are no (encrypted) messages in the HDCL data fields, these are almost surely some test transmissions.

It is worth noting that the QUEDRE string is clearly visible also in the output of any STANAG-4285 decoders by enabling the synchronous mode with 8N1 framing: probably the decoder will be Data Link aware so you will just see those repetitions and some other garbage due to the HDLC headers, addresses and CRC (pic. 8). Although the output text be consistent, this way is slightly misleading because you lose the knowledge of the Data Link.

pic. 8

http://www.interfacebus.com/HDLC_Protocol_Description.html 
http://www.synclink.com/html/hdlcmode.htm 
http://www.erg.abdn.ac.uk/users/gorry/eg3567/dl-pages/hdlc-framing.html 

MS188-110B, Appendix C/D scrambler

Both the High Speed Waveforms (HSWF) and Wide Band HF Radio waveforms (WBHF), described first time in the Appendices "C" and "D" from the standard MS188-110B, use the same scrambler. Modems operating over multiple discrete channels (Appendix F) also use this same scrambler since they use the waveforms from Appendix C. The scrambling sequence generator polynomial is:

x^9 + x^4 + 1

and is initialized to 00000001 at the start of each data frame, i.e. each 256 transmitted symbols (data block lenght is the same in both the two waveforms families). The length of the scrambling sequence is 511 bits,  computed as the maximal number of its states excluding the all-zeroes state (2^9 -1). For a 256 symbol data block with 4 bits per symbol, this means that the scrambling sequence will be repeated about 2 times, while for 6 bits per symbol just slightly more than 3 times, although in terms of symbols there will be no repetition (pseudo-random generator). In other words, the scrambler is designed to not have auto-correlation property, contrary to what happens for the MS188-110A scrambler, as we saw here, where the scrambler produces a periodic pattern 160 transmit symbols (480 bits, since the PSK-8 modulation) in length that at certain data rate speeds affects the value of ACF. 

I do not want reinvent the wheel here but only practice of analysis, so I just looked for a confirmation of this behavior (and described below) by analyzing the bitstream produced by a software-scrambler that I wrote in Lua language for both PSK-8 and QAM-16 modulations, in the latter case I also examined a real-world QAM-16 signal to verify the lack of possible signs/repetitions caused by the scrambler.

PSK-8 modulation

For PSK-8 data symbols (3200 bps and 4800 bps), the scrambling shall be carried out taking the modulo 8 sum of the numerical value of the binary triplet consisting of the last (rightmost) three bits in the shift register, and the symbol number.  A block diagram of the scrambling sequence generator is shown in pic. 1, in this illustration, three output bits are shown: this is the case for PSK waveforms.

Pic. 1

After each data symbol is scrambled, the generator shall be iterated (shifted) the required number of times to produce all new bits for use in scrambling the next symbol, so will be 3 iterations for PSK-8. Since the generator is iterated after the bits are use, the first data symbol of every data block (256 symbols) shall, therefore, be scrambled by the appropriate number of bits from the initialization value of 00000001.

pic. 2

The sw-scrambler writes the scrambled symbols, ready to be sent to the PSK modulator, to the "scrambler-output.txt" file. Examining this bitstream (pic. 2) a 768 bits period is revealed and this lenght is exactly the lenght of the 256 PSK-8 symbols data block: there is no evidence of 511 bits cycle due to the scrambler.

QAM-16 modulation
The data symbols for QAM modulations shall be scrambled by using an exclusive or (XOR) operation: sequentially, the data bits forming each symbol (4 for QAM-16, 5 for QAM-32, 6 for QAM-64 and 8 for QAM-256) shall be XORed with an equal number of bits from the scrambling sequence (pic. 3).

pic. 3 - scrambler for QAM-16 modulation

After each data symbol is scrambled, the generator shall be iterated 4 times to produce all new bits for use in scrambling the next symbol. As said, since the generator is iterated after the bits are use, the first data symbol of every data frame shall, therefore, be scrambled by the appropriate number of bits from the initialization value of 00000001. I used a constant data symbol value (0110) just to highlight the behavior of the scrambler.

pic. 4 - running the sw-scrambler for QAM-16 modulation

The sw-scrambler writes the scrambled symbols to the "QAM-16-scrambler-output.txt" file. Examining this 5000 symbols bitstream (pic. 5) a 1024 bits period is revealed. As in the case of PSK-8 scrambler, this lenght is exactly the lenght of the 256 QAM-16 symbols data block: also in this case there is no evidence of 511 bits cycle due to the scrambler.

pic. 5

real-world QAM-16 signal

pic. 6a - real-world MS188-110C App.D signal

pic. 6b - MS188-110C App.D, QAM-16 ACF

As expected, the ACF returns a 120ms period (pic. 6) that makes 288 symbols length frame at 2400 Baud. Since the data block for QAM-16 modulation is always 256 symbols, it follows that mini-probes are 32 symbols lenght (waveform n.8 from Appendix D TABLE D- XI):

After demodulating the QAM-16 signal with SA, it was then converted into an ASCII-bit file by using a simple HEX2BIN converter also written in Lua: the output file was then analyzed using a bit-flow processor tool. 
The analysis of the bitstream reveals a strong 1152 bits period that is exactly what is expected for the 4-bits symbols WBHF waveform (pic.7):

pic. 7

data-block: 256 symbols = 1024 bits
mini-probe: 32 symbols = 128 bits
frame (data-block + mini-probe): 288 symbols = 1152 bits
and in terms of symbols, there are no repetition caused by the scrambler (no auto-correlation property).

The same scrambling sequence generator polynomial x^9 + x^4 + 1 is also used in STANAG-4285 waveform (see Annex-A to STANAG-4285) but with a different inizialization vector (see the picture below and pic. 1):

and the results are obviously the same, the bit flow processor only detects the expected 768 bits period:

 

Unid MFSK 30 parallel tones + pilot tone and service tones

About the unid MFSK 30 parallel tones, reported in the previous post, since the frequency 16553.5 KHz/USB is operated by Japanese Military with their 4xFSK-2 system, it could be that this signal has that same source. 

The spectrum spreads a 2300 Hz bandwidth (pic. 1) and shows 30 tones ~70Hz spaced, in the lower part, after the strong pilot-tone, are visible four distinct tones transmitted at a lower power than the other 30: this could lead to think to a sort of "service" tones but it's a my guess (pic. 2). All the carriers are not modulated so the signal does not carry information. It's difficult to confirm the source (Japanese Mil.) as well as say if they are just testing a new system.

pic. 1

pic. 2

MS188-110 scrambler lenght and ACF (another test)

In a previous post I already talked about the way MS188-110 scrambler length affects the value of the ACF and only at certain data rate speeds, today I wanted to verify this matter but using a bitstream analyzer rather than SA. 

In order to do that, I needed to pick up transmission symbols from the output of the scrambler, just before the PSK-8 modulator: i.e. the "pure" symbols flow,  without any other addition such as mini-probes or preamble re-insertions. In other words, I needed a sort of reduced MS188-110 modem as shown in pic . 1

pic. 1 the reduced MS188-110 modem

Since this kind of reduced-modem simply doesn't exists, I decided for a software-implementation by writing a simple LUA [1] program, following the directions provided by the protocol [2] (see below): the program doesn't use bitwise operations but handle binary values as array of ASCII zeroes and ones  such as {101} for 0x101. Since the scope, FEC Encoder and Interleaver matrices have been omitted and replaced by a binary input file (formed of random binary data) and/or a constant binary data string. 

As said, known data sequences (mini-probes) and preamble re-insertions have also been omitted just to prevent ACF periods due to these patterns.

5.3.2.3.8 Scrambler.
The tribit number supplied from the symbol formation function for each 8-ary transmitted symbol shall be modulo 8 added to a three bit value supplied by either the data sequence randomizing generator or the sync sequence randomizing generator.

5.3.2.3.8.1 Data sequence randomizing generator.
The data sequence randomizing generator shall be a 12 bit shift register with the functional configuration shown on figure 6. At the start of the data phase, the shift register shall be loaded with the initial pattern shown in figure 6 (101110101101 (binary) or BAD (hexadecimal)) and advanced eight times. The resulting three bits, as shown, shall be used to supply the scrambler with a number from 0 to 7. The shift register shall be shifted eight times each time a new three bit number is required (every transmit symbol period). After 160 transmit symbols, the shift register shall be reset to BAD (hexadecimal) prior to the eight shifts.

As shown in pic. 1, the software output is a bitstream file called "scrambler-output.txt" that can be analyzed through a bit analyzer, and here we are: the result is shown in pic. 2., as it was to be expected, and incidentally reported in MIL 188-110 standard, the scrambler just produces a periodic pattern 480 (160 transmit symbols) bits in length!

pic. 2 - 480 bits (160 PSK-8 symbols) period

So, summarizing what has already been written in the cited post, this is why at low data-rate speeds (from 150 up to 1200 bps) the ACF analysis produces strong 66.67ms spikes (pic. 3): four contiguous groups of the pairs [mini-probe] + [unknown data] makes 160 symbols (pic. 4) and they are just in sync with the scrambler length! At lowest speed (75bps) there are no channel probe so the 66.6ms ACF is only due to the scrambler.

pic. 3 - MS188-110 ST 66.67ms ACF

pic. 4 - MS188-110 ST frame formation

[1] http://www.lua.org/

[2] http://hflink.com/standards/MIL_STD_188_110B.pdf

CIS FSK 100Bd/2000, Russian AF (in this case)

this signal has been caught on 14555.0 KHz on USB at 0840 UTC and 1240 UTC (the latter from KarapuZ). It's an FSK modulation characterised by 2000 Hz shift and speed of 100 Baud. The user is proved to be Russian and the Morse mode seen in the upper tone, decoded using MultiPSK software (pic. 3), suggests the Russian Air Force since the call REA4 belongs to Russian AF HQ Moscow. By the way, before the upper tone keyed the signal were not transporting data. More likely the Air Force is not the only user of the waveform.

pic. 2

pic. 3

examples of 2G-ALE followed by forward of data

MS188-141A + MS188-110A serial

MS188-141A + MS188-110A App.B

CODAN

MS188-141A + R&S GM2100

MS188-141A + voice comm

R&S ALIS + R&S GM2100

THALES Skymaster ALE + THALES TRC 177x serial modem

MS188-141A + STANAG-4285