STANAG-5066 CFTP messaging over MIL 188-110A

The transmission has been copied on 7788.0KHz/USB and consists of an emails exchange between an Algerian Navy ship (Rais Hamidou) and the Headquarter of French Navy in Toulon (COMTOULON).  Messaging is performed by Stanag-5066 CFTP protocol (Internet Messaging over ARQ) in half-duplex mode, carried by MIL 188-110A serial waveform (Fig. 1). Link is not negotiated/established using ALE but by means of voice-comms between the operators.

Fig. 1

Basically, CFTP (Compressed File Transfer Protocol), sometimes referred to as BFEM (Battle Force Email), is a file transfer mechanism and STANAG 5066 Annex F defines its use for message transfer. CFTP is specified to work over a point to point network - that is, to communicate between a pair of nodes, with both nodes able to send data.

Fig. 2 - S5066 bitstream synched on D_PDU SYNC

The CFTP protocol shall not be used for Formal or High Grade Military Messaging (military orders) and may be used for informal interpersonal e-mail only; anyway, I'm interested in the way the "boxes" travel and not in their contents.
In operation, when an email message is received at a S-5066 node (port 5066 on localhost), it is placed in an incoming mail folder (mail spool directory). The CFTP client, also called the Delivery Agent, removes the mail from this incoming folder and compresses the message and information about the message, e.g. size, id, recipients, etc. into a file which is then transferred to the destination node using the Edition-1 Basic File Transfer Protocol (BFTPv1)  PDUs which are then incapsulated into Data PDUs (Fig. 2) before to be sent to the HF modem. The structure of BFTPv1  PDU is shown in Fig. 3, along with the g-zipped file from the copied transmission.

Fig. 3 - BFTPv1 Protocol Data Unit Structure

Client Delivery confirmation is provided by the receiving node using the Message Acknowledgement, sent as the body-part (!) of a CFTP/RCOPv1 PDU:

Fig. 4 - BFTPv1 Message Acknowledgement Structure

The CFTP mail-file is built as specified in paragraph "F.14.6 Detailed Description of CFTP, ANNEX F to STANAG 5066 Edition 3":

Fig. 5 - CFTP email-file

Below, some details about the two STANAG-5066 nodes in the play:

node: EMMA046
tactical callsign: PI (papa india)
S-5066 address: 006.014.100.001 (official  NATO S-5066 address)
email domain: toulon.frenchnavy.fr
host: EMMA046 (Rockliffe SMTPRA 4.2.4)
site: COMTOULON Toulon, French Navy

node: BDSL472
tactical callsign: BD (bravo delta)
S-5066 address: 010.000.000.001 
(near-official NATO S-5066 address since the 010.001 block is assigned to Algeria)
email domain: rh.raishamidou.dz
host: BDSL472 (Windows NT 6.1; WOW64; rv:24.0)
site: corvette "Rais Hamidou", Algerian Navy

Rais Hamidou corvette, photo from wikipedia https://fr.wikipedia.org/wiki/Rais_Hamidou_(corvette)

The other labels in the emails headers offer some other informations:  

1) PC at COMTOULON run an SMTP daemon hosted on a Rockliffe MailSite based server (Rockliffe SMTPRA 4.2.4):
https://www.rockliffe.com/index.html

2) the "WOW64" in user-agent string means a 32-bit version of Internet Explorer is running on a 64-bit processor: so the ship is equipped with a 64-bit PC running Windows7 or Windows Server 7 (Windows NT 6.1)

By the way, it's interesting to note in the curious behavior of one of the two 188-110A modems and  precisely the one at the ship side of the link: the initial 2600Hz tone-beep is maintained for the duration of each single transmission time and does not affect the correct demodulation of the signal (Fig. 3)

Fig. 3

Although 188-110A is an auto-baud waveform, operators negotiate the speed and interleaver which will be used during the initial phase of the data transmission: this leads to think to inter-operability or deployment tests.

(due to confidentiality, wav file and bitstream are not avalilable. Sorry.)

CIS FSK 50Bd/250 136 bit (T-600-136)

Interesting signal from my friend KarapuZ: it's the quite uncommon 50Bd/250 136 Bit by some Cis networks, most likely CIS Navy. The waveform is similar to the well known T600 (50Bd 200/250Hz) but in this case the full period is 544 bits lenght, ie 4 x 136 bit frames.

Fig. 1

Fig. 2

https://yadi.sk/d/eESTZ-hN3HFu3R

splitting datagrams into LDLn Tx Frames

This post is added  to complete the previous one, so to study and verify the way a datagram is splitted to fit the length of the chosen LDLn Tx frame (n =32,64,96,…,512 bytes) and the mechanism used to fill the remaining room. This recording concerns a 139-byte lenght 'Citadel' encrypted datagram which is sent within five LDL32 PDUs, ie using 5 x 32 = 160 bytes: the protocol shall fill the extra 21 bytes of the last PDU with 0-value bytes (Fig. 1). 

Fig. 1

After the data link connection has been configured (FLSU/BW5), the sending station and the receiving station alternate transmissions in the usual STANAG-4538 manner: the sending station transmitting LDL PDUs containing payload data packets (BW3), and the receiving station transmitting ACK PDUs (BW4) each containing an acknowledgment of whether or not the data packet in the preceding LDL PDU was received without error. The sending station sends an EOM PDU (BW4) repeated as many times as possible to indicate to the receiving station that the the datagram have been delivered successfully and the link will be terminated (Fig. 2)

Fig. 2

For a better understanding of the way LDL protocol works the incoming datagram, let's see how the LDL PDU and BW3 are formed (quoting Annex C to NATO STANAG-4538).
If the Tx Frame buffer is clear, construct a new outgoing data packet in the following manner (Fig. 3):
1. Get the next data segment (the next PacketLength consecutive data bytes to be transmitted) from Tx Datagram buffer; place these in the Payload field of the data packet. If fewer than PacketLength data bytes remain to be transmitted, place these bytes at the
beginning of the Payload field; fill the remainder of the field with zero-valued bytes so that the Payload field contains a filled segment.
2. Construct a sequence number and write this value into the packet’s Sequence Number field.
3. Construct a control field with all 8 bits set to zero.
4. Place the data packet generated in steps 1-3 into the Tx Frame buffer.
5. Send an LD PDU containing the tx frame in Tx Frame buffer, using BW3 waveform.

Fig. 3

It's worth noting that:
- one LDLn Tx Frame, same as an LDLn PDU, consists of a single data packet of n-bytes (n=32,64,96,…,512); then LDLn delivers chunks of n-bytes at time (n=32,64,96,…,512);

- one HDLn Tx Frame, same as an HDLn PDU, consists of n-data packets each of 233 bytes (n=3,6,12,24); then HDLn delivers n-packets at time, each of 233 bytes (n=3,6,12,24).
 
The number of bits in a BW3 data packet is of the form 8n+25, where n = 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, or 512 (number of payload data bytes carried by each LDL protocol forward transmission);  the additional 25 bits of payload in each BW3 transmission are LDL overhead (17 bits Sequence number + 8 bits for reserved use). A 32-bit Cyclic Redundancy Check (CRC) value is computed across the payload data bits in the data packet, and is then appended to the data packet, then 7 flush bits having the value 0 are appended to the (8n+57) bits of the data packet with CRC to ensure that the encoder is in the all-zero state upon encoding the last flush bit.
So each BW3 data packet has a length of 8n+64 bits (8n+17+8+32+7): just the latest eigth bytes will be used for the analysis.

From theory to practice, the way the 139 byte datagram is splitted in five LDL32 Tx Frames can be verified by inspecting the  Sequence Number fields in the last 64 bits of the five BW2 data packets of the copied transmission (Fig. 4).

Fig. 4

Some aspects must be first considered:

a) the 8-bit reserved field is added after the CRC field and not after the Sequence Number, as specified in Annex C to STANAG-4538; I don't know if it's the modus operandi of the decoder;

b)  following the last bit of the Payload field-value, the bits of the Sequence Number field are transmitted starting with the least significant bit (bit 0) rather than the most significant bit (bit 16). Most likely it's the modus operandi of the decoder, as above;

c) as in the Sequence Number of HDL Tx frames (see the previous post), the bits 14-6 of the first packet in datagram contain the number of user bytes in packet -1 and this contrasts what specified in Table 7.1.4.1-2; it depends on the particular STANAG-4538 implementation?

c) the 'Packet Number', bits 5-0 in the Sequence Number field, indicates the position occupied by a data segment within a datagram. Since the datagram passed to LDL for delivery is an ordered sequence of up to 16,384,000 bytes (7,634,944 bytes for LDL), limiting the packets addressing to 6 bits could be fastidious in ARQ.

As expected, EOM and SOM fields (bits 16,15) show that the BW2 packet 0 contains the first packet of the datagram, while BW2 packet 4 contains the last packet of the datagram. Consequently, the Packet Number (bits 5-0) starts from 0 to 4: ie, the useful payload consists of five BW2 packets. Consider now the Packet Byte Count (bits 14-6): as already specified, this field show the number of user bytes in packet –1. Each of the first four packets contains 32 bytes (00001111 = 31) and the last packet contains 11 bytes (00001010 = 10). The outgoing datagram was then (4 x 32) + 11 = 139 bytes length.
During the traffic setup phase, the user process determines the number of data bytes per LDL PDU so as to deliver the user data efficiently. Once it is determined, every transmitted LDL PDU shall contain the same number of data bytes until the entire datagram has been delivered.

https://yadi.sk/d/PvOS5FbM3H96Wz
https://yadi.sk/i/UgFjnPTv3H97zz

 
 

housing of data packets in a HDLn Tx Frame

This recording is nothing special but rather a good example in order to study and verify the way the data packets are housed in an HDLn Tx frame and the mechanisms used to fill the unused room when the length of the outgoing datagram is less than the length of the Tx frame of the used HDLn version (n = 3,6,12,24 packets each of 233 bytes). Particularly, the recording concerns a 1889-byte lenght 'Citadel' encrypted datagram which is sent in one HDL12 PDU, ie using 233 x 12 = 2796 bytes: the protocol shall use the extra space of 907 bytes adding 0-value bytes and reinserting one or more data packets (Fig. 1).

Fig. 1

For what concerns the on-air signal (Fig. 2), the HDL12 PDU is trasmitted within a BW2 waveform across an already-established HF link (FLSU/BW5) and is folowed by three EOM PDUs (BW1) sent by the transmitter station meaning that entire datagram has been successfully delivered. The HDL12 PDU is acknownledged by an ACK PDU (BW1) transmitted by the receiver station. The logical link will be terminated by a couple of FLSU exchanged  between the two stations. The transmission was copied on 8327.0 KHz/USB.

Fig. 2

For a better understanding of the way HDL protocol arranges the extra 917 bytes, let's see how the HDL PDU and BW2 are formed (quoting Annex C to NATO STANAG-4538).

For each of the first data packet positions in HDL TxFrame buffer (12 positions in this case), if the data packet position is empty, construct a new outgoing data packet in this position:

1. Get the next data segment (the next 233 consecutive data bytes to be transmitted) from
TxDatagram buffer; place these in the payload field of the data packet. If fewer than 233 data bytes remain to be transmitted, place these bytes at the beginning of the payload field; fill the remainder of the field with zero bytes.

2. Construct a sequence number value and write this value into the packet’s Sequence Number field.

 3. If TxDatagram buffer is completely emptied of payload data before all packet positions in TxFrame buffer have been filled with new packets, fill the remaining packet positions with repetitions of packets already residing in other positions of TxFrame buffer. The HDL transmitter is at liberty to select packets from the current datagram to repeat as it pleases; the HDL receiver must inspect the sequence number of each packet received without errors, and use this information to discard duplicate packets. 

4. Send an HDL PDU containing the tx frame in TxFrame buffer, using BW2 waveform.

Fig. 3 - formation of the HDL12 Tx fame for this example

The 0-value bytes and the packet repetions are well visible in Fig. 1.

During the construction of BW2 waveform structure, a 32-bit Cyclic Redundancy Check (CRC) value is computed across the 1881 payload data bits in each data packet (1864 bits of user data, plus a 17-bit Sequence Number added by the protocol), and is then appended to the data packet. The seven encoder flush bits with values of zero (Flush) are then appended to produce an Extended Data packet of 1920 bit length that will be used for FEC, frame formation, symbol formation, and so on. 

From theory to practice, the way the data packets are housed in TxFrame buffer can be verified by inspecting the  Sequence Number fields in the last 56 bits of the first nine BW2 extended data packets of the copied transmission (Fig. 4).

Fig. 4 - the last 56 bits (Sequence Number + CRC + Flush) of the copied BW2 Extended Packets

As expected, EOM and SOM fields (bits 16,15) show that the positions 0 and 9 of the TxFrame buffer are both filled with the first packet of the datagram, while position 8 of the TxFrame buffer is filled with the last packet of the datagram. Consequently, the Packet Number (bits 5-0) starts from 0 to 8: ie, the useful payload consists of 9 packets (note also that positions 0 and 9 have the same Packet Number).
Consider now the Packet Byte Count (bits 14-6): this field show the number of user bytes in packet –1. The first eight packets, positions 0-7, contain 233 bytes (011101000 = 232) and the last packet, in position 8, contains 35 bytes (000100010 = 34). The outgoing datagram was then (8 x 233) + 35 = 1899 bytes length.

One could say "since the datagram length is 1899 bytes and fills 9 data packets, why don't use 1 HDL6 PDU + 1 HDL3 PDU?". It would be impossible.
During the link setup phase, the session manager process determines the number of data packets per HDL PDU (3, 6, 12, or 24) shortening the HDL PDU whenever the entire datagram is short enough to fit within the shortened PDU. Once it is determined, the number of data packets per HDL PDU for the current datagram delivery remains unchanged, ie every HDL PDU contains the same number of data packets until the entire datagram has been delivered (so: 3 x HDL3 PDUs, or 2 x HDL6 PDUs, or just 1 HDL12 PDU... btw more PDUs means the more turnaround time).

https://yadi.sk/d/wcvQBJfr3H75uc
https://yadi.sk/i/j0nqeWiC3H75x9 

an LDL160 transfer with nine retransmissions

this is a copy of an 10 x LDL160 transfer, heard on 8327.0KHz/USB, with up to nine retransmissions of the same 'Citadel' encrypted  data packet (Fig. 1). Since the LDL forwards are followed by ACKs from the receiver station, although they are faintly visible, the nine retransmissions could be due to repeat requests.

Fig. 1

The analysis of the last 64 bits of each BW3 bursts shows a fixed structure in the Sequence Number field: if the reps had been in the datagram passed to the LDL protocol then the sequence number would have been incremented, moreover the EOM and SOM fileds (bits 16,15) are both to 1 in all the packets meaning that this is the only packet in the sent datagram  (Fig. 2)

Fig. 2

As said, since the ARQ ACK mechanism of xDL protocols, it could be that the nine retransmissions are requested by the receiver (bad channel conditions at receiver side?) but it could also be a technique to improve the reliability of the transmission: transmitting the same information more than once (each time encoded using alternate FEC codes) during a particular link increase the probability that user data will be received correctly.

https://yadi.sk/d/_SCU3U073GzPf9

a possible EMCON retransmission mode for Multicast Data Link protocol ?

This is an MDL-N (Multicast Data Link with NACKs) transfer received on 10958.0 KHz/USB. The transfer begins with an FLSU Point-to-MultiPoint call (BW5) followed by four forward transmissions consisting of LDL288 DATA PDUs (BW3); the ending BW4 burst is the LDL EOM PDU informing the receiving stations that the entire datagram has been transmitted. In MDL-N each forward transmission is followed by a pause, during which receivers that were not able to decode that transmission emit a very robust pseudonoise PSK symbol sequence to request retransmission: in this sample there is no NACK PDUs issued by the receiver stations.

At a first glance, one could say that the incoming datagram at the sender has been splitted in four LDL 288-byte data packets (unless the padding bytes in the last packet) which are then forwarded using four BW3 bursts ...but things are not properly in this way.
Indeed, looking at Figures 1-2,  it's easy to notice that packets #1 and #2 are exactly the same, as well as the packets #3 and #4 (275 bytes length). 

Fig. 1 - packets #1, #2

Fig. 2 - packets #3, #4

This means that each packet is sent twice before to clear the TxFrame buffer, so that the receiver stations have two copy of the same message, each obtained by adding the packets #1, #3 and the packets #2, #4 (Fig. 4)

Fig. 4 - the received entire datagram

(note that the encryption is applied before the datagram is segmented).

According to these observations, I try to give an explanation that makes sense to that scenario.
For a reliable transmission of the message, each packet is always sent twice (at leats in this sample): a station that decodes the entire error-free message can deliver the message and discard the copy. Otherwise it must process the copies and attempt to correctly decode the message. Summarizing: each packet is transmitted n-times instead of n-retransmissions of the complete message.  This mechanism could contrast with the chance of use ACK/NACK PDUs, but - other than adds redundancy - it could be also a good solution in case of receiver stations which must remain in EMCON mode (radio silence) for extended periods and consequently can't send NACKs neither delayed acknowledgements.  
The same behavior has been observed also in some LDLn transfers, but here the retransmissions could be requested by the receiver station.
Most likely, to increase the reliability a sort of 'retransmission counter', or a 'configurable setting', indicates the number of times each packet shall be retransmitted (repeated) or not during a particular link , according to the established xDL PDU for that transmission and the conditions of the channel, no matter if the packets have been ACKed or not, to increase the probability that user data will be received correctly.

As pointed out, this is a my thought, a supposition based on what I see: I do not know if it's  just a coincidence or a sort of implementation of STANAG-4538 since I have not found confirmations in the documentation that is publicy available in the web. Further recordings are needed in order to better understand this kind of scenario. By the way, your comments are welcome.

 https://yadi.sk/d/J8mreWoP3Go2ip

xDL PDUs, BW2 & BW3 waveforms, and Harris 'Citadel' encryption

I spent some time to understand where and when the 'Citadel' encryption is applied to a message in the STANAG-4538 xDL implementation by Harris, eg in the RF-5800H transceiver, regardless of whether the message came from STANAG-4406 P-MUL or HMTP/CFTP and at least (!) in my copies.

HDL and LDL protocols exist in different variants, and a number n (eg xDLn) specifies the size of one forward transmission. For HDL the number n (24, 12, 6, or 3) should be multiplied by 233 bytes plus a 17-bit sequence number added by the protocol to give the total number of bytes in one forward Tx frame.  If the data segment is of length less than 233 bytes,  a sequence of null data bytes (of value zero) is appended to the data segment so as to extend it to length 233 in constructing the data packet.
In case the datagram is emptied and not all data packet positions have been filled with new packets, the remaining data packet positions are filled with repetitions of packets already residing in other positions. The HDL transmitter is at liberty to select packets from the current datagram to repeat as it pleases; the HDL receiver must inspect the sequence number of each packet received without errors, and use this information to discard duplicate packets.  Note: Whenever a packet is retransmitted, it is always placed in the same packet position within the Tx frame that it occupied in the previous transmission.
Fig. 1 shows the formation of two HDL3 forward transmissions.

Fig. 1 (not in scale)

For LDL, the number n (32, 64, 96,…,512) gives the number of bytes explicitly in a tx frame plus 25 bits due to LDL overhead, ie 17-bit sequence number plus 8-bit data reserved for future use. As for HDL, the tx frame length of the LDL protocol has discrete values, so that a data segment will be padded with 0 value bits if ist length is less than the packet length established for the current LDL transfer (32, 64, 96,…,512). Fig. 2 shows the formation of two LDL512 forward transmissions.

Fig. 2 (not in scale)

Now assume that a STANAG-4406 message shall be encrypted and transferred. P-MUL protocol segments a message into UDP/IP packets with a maximum length of the IP packet of 1500 bytes. If the length of the message to be transferred is 15 kbytes, a total of 11 IP consecutive packets are required for the message transfer. Consider now how the STANAG-4538 protocols operate when transferring one IP packet. At the arrival of the first IP packet at the sender station, a link to the destination node is set up by FLSU PDUs. Then, a 1500 byte datagram containing the first IP packet is transferred for example by using two HDL6 Data PDUs or three LDL512 Data PDUs, each related to one segment of the datagram and followed by an ACK in the reverse direction. 

The Citadel engine works on each datagram before to construct the xDL PDU, in case of the incoming datagram matches the length established for the current xDL transfer then the encryption could be seen as directly applied to the xDL PDU. Since the encryption adds overhead bits, the length of the datagram segments is chosen accordingly.

encryption in a HDL formation

The insertion of the encryption is more clear looking at xDL bitstreams from the real world. An HDL12 transfer is shown in Fig. 3: the presence of the Citadel "business card" is well visible just at the beginning of the PDU (Fig. 5).

Fig.3 - Citadel encrypted 1 x HDL12 transfer

Fig.4 - Citadel encrypted 1 x HDL12 transfer

An LDL32 transfer is shown in Figs. 5,6 : the presence of the Citadel pattern is well visible just in the first packet of the datagram.
 

Fig. 5 - Citadel encrypted 2 x LDL32 transfer

Fig. 6 - Citadel encrypted 2 x LDL32 transfer

Looking at Fig.2, the correspondence 1 LDL PDU = 1 data packet could wrongly lead to think to an encryption applied on data packet basis: the encryption insertion is more clear looking at HDLn, where 1 HDLn PDU = n data packets (Figs. 3,4).

Now look carefully at Fig. 5: each packet is indicated with its index followed by the dimension of the data segment, 32 bytes in this case (LDL32). That said, why five 8-byte rows, ie 40 bytes, are printed ? The same incongruity is verifiable in HDL packets: 240 bytes printed instead of the expected 233 (fig. 7)
 

Fig.7

LDL use burst waveform BW3 to transmit its PDUs. The number of payload bits is of the form 8n+25, where n = 32, 64, 96,..., 512 and the additional 25 bits are the overhead consisting of 17 bits of the sequence number + 8 bits for future use. A 32-bit Cyclic Redundancy Check (CRC) value is computed across the payload data bits in the data packet, and is then appended to the data packet. Then, 7 flush bits having the value 0 are appended to the (8n+57) bits of the data packet with CRC. In the example of Fig. 7 (LDL32) we obtain (8 x 32) + 57 + 7 = 320 bits that make the 40 bytes.

For what concerns HDL PDUs, these are transmitted by BW2 burst waveform. Each data packet is composed of a 1864 bits data-segment (233 bytes) + 17 bits of the sequence number. A 32-bit CRC value is computed across the 1881 payload data bits in each data packet, and is appended to the data packet. Then, seven encoder flush bits with values of zero are appended to the 1913 payload and CRC bits of each data packet to produce an Extended Data Packet (EDataPkt) of 1920 bits or just 240 bytes.

Juts another way to say that HDL is not BW2, and LDL is not BW3: xDL are protocols while BWn are waveforms.

For what concerns MDL protocol, it seems that the Citadel encryption is applied on the incoming datagram before its segmentation (Fig. 8).

Fig.8 -  MDL transfer using 3 x LDL32

Behavior of STANAG-4538 depends on vendor implementation as well as on its configurable parameters, so this post does not aim to show a sort of  fixed rule but just some cases of  transmissions that happened to hear.