Overview: The Club Log Gateway

What is it?

The Club Log gateway is a tool developed by G7VJR to enable real-time uploads from three popular networked logging software tools, DXLog, N1MM and Win-Test. It is designed for expeditions. It replaces other plugins that upload to Club Log with a more efficient and reliable solution. It is also the only real-time solution that supports log edits and log deletions (not just adding QSOs). 

Where can I get it?

The gateway is a command line tool. Inside the downloaded ZIP is an executable which you run from the command line. 

Notes: 

1. Linux (ARM) can be used on recent Raspberry Pis
2. The gateway will alert you when new builds are available.
3. More instructions are provided below, under "How to download and run the gateway".

Who is it for?

Expedition operators and teams often use advanced logging tools, notably Win-Test, N1MM and DXLog. It will also work any tools that faithfully and accurately implement the same protocols. For example, RumLog and WSJT-X work, as they implement the N1MM protocol correctly.

These are networked, multi-operator tools. When multiple logging computers are in use, every QSO is mirrored immediately to the other computers on the network, and all of the operators can see everything in real-time. These tools are also favoured by single operators because of their refined features and efficient, contest-style logging interfaces. 

For more information see: https://n1mmwp.hamdocs.com (N1MM), http://www.win-test.com (Win-Test) and http://dxlog.net (DXLog).

If you use one of these logging tools, and want to upload to Club Log in real-time, the Club Log Gateway is for you.  However, if you're logging DX at home, you don't need this tool. Just carry on using your normal software as usual.

What does it do?

The Club Log Gateway (or just 'gateway') is a small command-line tool that is designed to run on a Windows, Mac or Linux computer and listen to the network protocol of the logging software. It stores a record of the QSO (including handling edits and deletes) in a database file that is stored in the folder alongside the tool itself. 

This database keeps a carefully managed check on what is happening. Then, the gateway periodically sends the QSOs in its database to Club Log (and will keep trying, if there are any problems getting through). Once it's succeeded, it will not need to resend those QSOs unless they get edited or deleted, in which case again it knows how to get the changes back to Club Log.

This is achieved in near real-time, including via slow connections such as satellite phones, using an extremely efficient and lightweight protocol that saves over 90% of the normal overheads of ADIF uploads. This means that real-time uploads are realistic, and not too hard to achieve. The payload is about 500 bytes for 35 QSOs. Using the gateway, you can easily push QSOs into Club Log within a few seconds.

Here's a diagram of how it works at the most basic level:

The gateway is easy to use, and designed for robustness with specific functionality on both ends of the connection to make it as bulletproof as possible. If you want to upload to Club Log automatically, in real-time or nearly real-time on an expedition, you should, ideally, use this tool instead of doing it any other way (even if bandwidth is not a big concern).

The gateway has some advanced concepts which will be better suited to the largest expeditions, but you will need some additional skills if you use those features. I've indicated a notional "difficulty level", below. It starts simple and gets harder - so just stop when you've got to what suits you best.

How to download and run the gateway

First, download the right binary for your computer (see above). In the documentation that follows, I'll use "clublog-gateway" rather than the full filename for your platform, so you will need to ensure you use the right command each time. For example, on Windows, when I refer to "clublog-gateway", you'll use clublog-gateway-windows_amd64.exe.

This is a command line tool. It is purely text-based. To run the program, open a command line prompt or terminal. 

Tip: On Windows 10, I use Control+R and type cmd then press enter to open a terminal. On a Mac, I use the /Applications/Utilities/Terminal.app application (open it using Finder).

Run it with -h to get help (the command line options are also explained on this documentation page, further down).

Windows:

clublog-gateway-windows_amd64.exe -h

Linux and Mac (all flavours):

Make sure the binary is executable (chmod +x clublog-gateway-linux_amd64).

Either provide the full path, or run the tool in the folder where you want it to operate.

./clublog-gateway-linux_amd64 -h

or like this:

/home/g7vjr/clublog-gateway-linux_amd64 -h

Consider making the gateway load automatically. For example, on Windows you could put the command into a Batch file (.bat) that runs at startup, or on Linux, make a cron job that runs it after a reboot. You can also use systemd on Linux.

Tip: on Linux, ensure you have sufficient file descriptors available (eg.. in systemd use

LimitNOFILE=100000).

Authenticating with an Application Password

You will need an App Password The gateway needs to authenticate with Club Log. To do this, it uses a Club Log Application Password when it connects. You can get one by logging into your Club Log account, and going to Settings (top menu), then App Passwords.

An App Password looks like this: 407-Xl908-Rm690-Xx661-Wi881-Bg822-Hn

Please don't try and use your normal Club Log password. It won't work - you must use an App Pasword, and the callsign you use must be one that is available in your account.

Basic use: one logging PC (single operator)

Difficulty level: low

Your laptop will be the logging computer as well as providing the internet connection for the QSOs to be uploaded to Club Log.

Sending the messages to the LAN when there is just one computer is unnecessary. The loopback (localhost) interface is perfect for our needs. The loopback IP address is 127.0.0.1. Just make sure your logging software is broadcasting to 127.0.0.1. This might not be the default, as some loggers assume they will broadcast on their LAN IP address. 

Here's an example command to run. Run it in a folder that you can read and write to. Keep it running at all times! If you restart it, make sure you run it in the same folder because there will be a small database file that persists between the restart.

clublog-gateway -mode=listener -ip=127.0.0.1 -callsign=[expedition-callsign] -password=[clublog-app-password] -rate=30

When the gateway is running, you will see information printed to your screen each time QSOs are added, edited, deleted or uploaded. If there are any signficant issues, they will also be logged to the screen. Wherever possible, the gateway will work through problems and stay running. You should try and keep an eye on it so you are aware if something looks wrong. Remember, if it's not running then the QSOs are not being recorded.

About the -rate flag:

The -rate=30 flag means the gateway will upload new QSOs at most once every 30 seconds. By batching QSOs together like that, you can economise on transfers. Compressing all the QSOs made in 30 seconds into a batch is very efficient. For comparison, if you set the rate to -rate=1, uploads are close to instantaneous, but you'll generate a lot more round-trips with smaller transfers each time. 

The gateway only connects if there is something to transfer. The gateway also has a "dead time" feature, which provides a period of cooling off on new QSOs logged, and is set with -deadtime=[number of seconds]. It defaults to 10 seconds.

Intermediate use: several logging PCs (multi operator) all on the same LAN

Difficulty level: low

One of the laptops, ideally one which will be running all the time, runs the gateway. The gateway listens to the local LAN for broadcasts by the logging software. The LAN should ideally be a wired network (because UDP packets often get lost on Wifi networks).

I will use this network in the command line example. If your LAN IP is 192.168.86.30 (and it's a regular class C network, ie. it has a netmask of 255.255.255.0) then your broadcast address is 192.168.86.255. That is the address to enter in your logging software. The Club Log gateway can be run without any special flags, because it tries to listen on all available networks automatically.

The logging software you use must be carefully set up on each client to use the right broadcast address, and it must be the same everywhere on all the computers.

Tips: Take care if you're on a guest hotel network or something shared with others, because the chances are it will cause problems. Your best solution is a wired network or if you can't avoid it, a dedicated wireless network that is used solely for your logging clients. Remember, also, that all of your logging PCs will probably need to have their firewall disabled (and all the automatic software updates probably ought to be temporarily disabled) so that packets can actually move around the network freely. Your ideal situation is a network that is dedicated to amateur radio use by your expedition.

Now, on the computer you have selected to run the gateway, use this command:

clublog-gateway -mode=listener -callsign=[expedition-callsign] -password=[clublog-app-password] -rate=30

This method collects all the QSOs it hears on the network and then sends them to Club Log once every 30 seconds.

Tip: This should not normally be necessary, but if you need more control over where the gateway listens, you can specify an interface using -ip=192.168.86.30 (which makes the gateway only listen on the network interface where you have raised the IP address 192.168.86.30). Make sure you specify an IP address that is available on the computer running the gateway, and that the logging software is also broadcasting to that network. 

Advanced usage: multiple logging PCs, avoiding any use of UDP broadcasts (works properly on Wifi)

Difficulty level: moderate

The Club Log gateway client can be started with a different mode, making it into a hub. 

This mode exists to solve a problem that the logging programmes use UDP broadcasts. When a message is broadcast on UDP, it is not a sure thing that any other clients on the network will hear it. The packet is fired and forgotten. That can result in lost QSOs. 

Packet loss is a very real problem, especially on Wifi networks. However, when a message is sent with TCP it will arrive at its destination, even if it has to be resent or broken up and reassembled along the way. TCP's guaranteed delivery is obviously desirable when we're logging QSOs, because every QSO is valuable. 

In this mode of operation, the Club Log gateway runs on each local logging PC's own network interfaces (as necessitated by the logging software's own functionality to coordinate more than one logging station). It then sends them to a hub using TCP. The hub deals with the Club Log uploads.

The hub is generally on a separate computer (perhaps a Raspberry Pi), but it can run on one of the logging PCs if you are certain that the PC will be 100% available and running at all times. Most logging computers get restarted a fair bit, and if the hub isn't running, QSOs won't be recorded. If that happens, you'll have to merge them with ADIF later. It's worth planning a separate computer if you can!

In this example, the hub's IP is 192.168.86.192. I will use this in the command line example.

A hub is started with different command line options. Now you set the mode to be "hub" instead of "listener", and provide the listener's IP to bind to:

clublog-gateway -mode=hub -callsign=[expedition-callsign] -hublistenip=192.168.86.192 -password=[clublog-app-password] -rate=30

The default TCP port is 8432. If for some reason this is not suitable, you can change it using hubport=[some other port]. Keep the hub running at all times.

Once you have a hub running, your clients need to be told it exists. This is an example you would run locally on a logging computer (remember, each logging PC runs this command, and keeps it running at all times):

clublog-gateway -mode=listener -callsign=[expedition-callsign] -hubip=192.168.86.192

Notice that the listener doesn't need Club Log credentials any more. It will not be communicating with Club Log, only with the hub!

Tip: We can't avoid using the LAN/WiFi network, since the logging software on each PC relies on that to communicate. But, by running a listener on every PC, we can ensure we don't miss any broadcasts made via UDP.

Advanced usage: multiple logging PCs, no UDP, multiple hubs, fault-tolerant

Difficulty level: Experts only.

Any client can send its messages to a hub. This also includes hubs themselves, which can send their messages on to other hubs.  Hubs can also reflect messages between each other. This is to allow for fault-tolerance.

Here is an example of how you could use the gateway to create a resilient solution, with two hubs that are keeping each other informed of changes. 

The first hub is listening on 192.168.86.200, and the second hub is listening on 192.168.86.192. Both of them are syncing to Club Log once a minute. The Logging PCs are listening to UDP traffic on all of their network interfaces.

Logging PC #1

Sends logs to the first hub: 192.168.86.200

clublog-gateway -mode=listener -callsign=[expedition-callsign] -hubip=192.168.86.200

Logging PC #2

Sends logs to the other hub: 192.168.86.192

clublog-gateway -mode=listener -callsign=[expedition-callsign] -hubip=192.168.86.192

Hub #1:

Listens on 192.168.86.200 and repeats everything it hears to Hub #2 at 192.168.86.192

clublog-gateway -mode=hub -callsign=[expedition-callsign] -hubip=192.168.86.192 -hublistenip=192.168.86.200 -password=[clublog-app-password] -rate=60

Hub #2:

Listens on 192.168.86.192 and repeats everything it hears to Hub #1 at 192.168.86.200

clublog-gateway -mode=hub -callsign=[expedition-callsign] -hubip=192.168.86.200 -hublistenip=192.168.86.192 -password=[clublog-app-password] -rate=60

In this scenario, the hubs repeat the messages they received to each other, resulting in a stronger solution as there are now copies on both hubs at all times. You can add more hubs if you need to (basically, chain them together in series - or even loop them into a ring). Hubs have features built-in to prevent them creating infinite loops or packet storms.

As you will have noticed, either of the hubs can fail at any time. What you really want is that the Logging PCs don't depend on one hub or the other, but can use either of them. So next, we need to use HAProxy (High Availability Proxy, see http://www.haproxy.org). This is a Linux tool. I'd recommend running it on a Raspberry Pi. You are going to need to read at least some of the docs for HAProxy. It's a simple, well-known tool and I won't repeat how it works here. If you've never used HAProxy before, then please set aside a sensible amount of time to get familiar with it before you decide to go down this route.

HAProxy listens on one IP address and then relays (proxies) to a pool of backend servers. Since we're using TCP to talk to the hubs, and one hub is as good as any other, all you really need is a pool with at least one hub actually working.

Here's an example haproxy.conf for Club Log:

frontend clublog

    bind *:8432

    mode tcp

    timeout client  5s

    default_backend hubs

backend hubs

    mode tcp

    balance leastconn

    timeout server  5s

    server hub1 192.168.86.192:8432

    server hub2 192.168.86.200:8432

Reminder: the Club Log gateway uses TCP port 8432 for hub communications by default.

HAProxy Listens on port 8432 for client traffic and load balances it between available backend hubs. In the configuration example, HAProxy is bound to all available interfaces. You'll need to know the IP of the Raspberry Pi and then that's the IP that the client logging PCs should connect with.

If HAProxy is running on 192.168.86.100, then make the client logging PCs talk to HAProxy on that address rather than talking to the hubs directly:

clublog-gateway -mode=listener -callsign=[expedition-callsign] -hubip=192.168.86.100

This solution avoids any dependence on UDP, and ensures you have 100% availability of the upload gateway even if one of the hubs is offline. You can also make the HAProxies fault-tolerant (by using more than one), and floating the HAProxy IP address using Corosync/Pacemaker. If this seems excessive, unfortunately it's just how things are when you seek perfect redundancy.

Tips: If a hub has been offline and restarts, just start it again and let it carry on as usual. Any overheads involved in untangling changes will be sorted out by Club Log on the remote end as expected. If you really want to completely reset it, stop the gateway, delete the .db file in the gateway's working directory, and start it again. This should not be necessary unless the database file has been damaged. You can also copy the .db file from another hub (but you will need to stop the donor hub before copying the file away, to ensure you get a consistent copy). There is nothing specific to one hub stored in the database.

Embedded NTP (time) server

It's very frustrating when all the PC clocks on your network are different. This is much more likely to be an issue if they are not connected to the internet, as they will not get time updates from their usual sources. Ideally, however, you'll keep one of them accurate and the object then is to get all of the other client PCs to use that reference.

To help with this, the Club Log gateway client provides an NTP server on the default UDP port 123.

You can enable the NTP server by using -ntpip=[ip address] at the command line. Since this is a privileged port number, when you run the gateway, it will need to be a privileged user (root on Linux or Mac, or an Administrator on Windows). You'll see an error about this if you run the gateway without the necessary privileges.

Once the NTP server is up, you can set the other PCs on the network to use it as their time server. Use its IP address in your NTP client (there is one built into every OS - you'll need to Google it if you're not sure where!). Once that's done, the client PCs will sync to the time of the server. A hub makes a good place to run the NTP service, especially if it has a connection to a real time source such as GPS, or the internet (even if only sporadically).

Running on a Raspberry Pi

A Raspberry Pi makes the ideal platform for running the gateway, as it can be put onto your network alongside the logging computers, but out of the way so that users do not inadvertently disrupt the process (which must run all of the time).

A very helpful walk-through for setting up a Raspberry Pi to run the gateway is available here: https://gist.github.com/WheezeNL/294b34aceb6d5bd2b3fef6fd89a9762a

Don't forget to allocate more file descriptors, which is not mentioned. e.g. in systemd in the [Service] section, add 

LimitNOFILE=100000.

Thank you to Lennart, PA3L for his contribution!

Command line options

  -callsign string
        Expedition callsign
  -deadtime int
        Delay uploads until QSOs are this number of seconds old. eg. -delay=60 for one minute. (default 10)
  -debug
        Debug mode (for use if requested by Michael, G7VJR). Verbose UDP logging and raw internal variables and structs are dumped to the console. eg. -verbose=true to enable.
  -dxlogport int
        Port number for DXLog UDP packets (default 9888)
  -freq
        Whether to include frequencies (increases upload payload). eg. -freq=false to disable.
  -hubip string
        Repeat UDP traffic heard to a hub at the specified IP address. If unset, this feature is bypassed.
  -hublistenip string
        The local TCP address to listen on (defaults to all)
  -hubport int
        Communicate with hubs on this TCP port (default 8432)
  -ip string
        Bind the listener to the network interface having this IP address, eg. 127.0.0.1 (defaults to all interfaces)
  -mode string
        Choose a mode of operation [listener|hub|dump] (default "listener")
  -n1mmport int
        Port number for N1MM UDP packets (default 12060)
   -ntpip string
        IP address to raise an NTP (time) server. Disabled by default. eg. 
          -ntpip=192.168.0.50.
  -password string
        Club Log App. password
  -rate int
        Minimum number of seconds between log uploads (omit or set to 0 to disable)
  -secure
        Whether to use HTTPS. Increases payload size. eg. -secure=true to use HTTPS.
  -wintestport int
        Port number for Win-Test UDP packets (default 9871)

Other Notes and Tips

The Club Log gateway listens for all of the supported message types (DXLog, N1MM, Win-Test). You don't need to declare which one you're using.

On Win-Test or DXLog, to delete a QSO go back and change the callsign to the expedition's own callsign. On N1MM, use the delete option in the right-click menus.

The only modification to a QSO supported is to change the callsign. If you change the band, mode, etc. then depending on the logger this will delete the old record and insert a new one, or just create a new record for the QSO in addition to the old one. The limitations of the messages passed to the gateway will apply in these cases.

The Club Log hubs never send anything on UDP. They don't act as a repeater for UDP traffic. 

If you want to you can send hub traffic over the internet - it's a gzip compressed version of the messages that were heard on UDP, but only the events that actually involve QSOs being logged, edited or deleted. I am not sure there is a good reason to do it, but it is supported.

If you want to reset a gateway client completely, stop it and delete the .db file in the working directory, then restart it. All QSOs in the local database are lost, but subsequent logging activity goes ahead as usual.

The 'dump' mode mentioned in the command line docs is to run a single export of the local database (without staying running to listen for UDP or be a hub). This is useful if you have restored or copied .db files and want the gateway to quickly push them to Club Log.

If you realise that your gateway hasn't been running for a while, or you started operating and forgot to load the gateway, just start it up as usual - and carry on logging - then, when it's convenient, export an ADIF from your logging software to Club Log via the normal web site upload tools. Club Log will handle merging this in for you.

The 'deadtime' flag is designed to combat anyone depending upon real-time logs to complete a QSO. The QSOs are uploaded according to the upload rate, but QSOs must be minimum number of seconds old. The default is 10s, but you can disable this feature by setting it to zero. A delay can be enough to alert the DXpedition operation that the QSO should not be logged as the DXer is not hearing them and is calling blind.

On Windows, make sure you accept any firewall exceptions (or switch your firewall off). Windows automatically prompts for firewall exceptions when a programme tries to open a port, so you should see a prompt like this. Just click "Allow access":

Planned Features

• The ability to import an ADIF file (instead of, or as well as, listening to UDP broadcasts)
• The ability to export an ADIF file.
• Support for other expedition logging software if justified by demand.

Not planned: support for home logging software. There is no need to use this gateway at home! Just use one of the pre-existing real-time upload features already available (and widely implemented). See: https://clublog.freshdesk.com/support/solutions/articles/53200-which-logging-software-supports-realtime-uploads-