Sunday, 8 September 2013

Almost Real Time DSC Logs

I've been tinkering with some Linux Shell scripting and using a script to split my current DSC Decoder log file into separate chunks, and then to make them accessible from the web.

Every 10 minutes the script will copy the current day's logs, and then make available the logs containing:

1) Unfiltered "Decoded" and "Formal" logs from 2MHz and 8MHz

2) Filtered Logs from both bands:


  • GMDSS Training - Dover (Snargate Radio) : MMSI 002320204 - 2MHz and 8MHz
  • Messages sent FROM a coast station - 2MHz and 8MHz
  • Messages sent FROM a ship station - 2MHz and 8MHz
  • Messages that are NOT TEST Calls - 2MHz and 8MHz
  • Messages TO or FROM "Shetland Coastguard" - 2MHz
  • Messages TO or FROM "All UK Coastguard" - 2MHz
  • Messages TO or FROM "All USA Coast Stations" - 2MHz and 8MHz
  • Messages of type "DISTRESS" - 2MHz and 8MHz


The files are updated every 10 minutes, and a new set of files is started at midnight UTC each day - so the only data available is for the Current Day.

To access the data :

https://dl.dropboxusercontent.com/u/3551430/grabber.html

The Shell Script which splits and formats the files (with my first ever use of the Linux "sed" command) is below

#!/bin/bash
DATE=`date -u +%y%m%d`

infile1=/home/gm4slv/Dropbox/Logs/DSCDecoder/2mhz/dscdecoder$DATE.log
infile2=/home/gm4slv/Dropbox/Logs/DSCDecoder/2mhz/dscformal$DATE.log
infile3=/home/gm4slv/Dropbox/Logs/DSCDecoder/8mhz/dscdecoder$DATE.log
infile4=/home/gm4slv/Dropbox/Logs/DSCDecoder/8mhz/dscformal$DATE.log


# Put new lines on the log files
sed 's/$/\n/g' $infile1 > /home/gm4slv/gm4slv_2mhz_dsc.txt
sed 's/$/\n/g' $infile2 > /home/gm4slv/gm4slv_2mhz_formaldsc.txt
sed 's/$/\n/g' $infile3 > /home/gm4slv/gm4slv_8mhz_dsc.txt
sed 's/$/\n/g' $infile4 > /home/gm4slv/gm4slv_8mhz_formaldsc.txt


# Look for Dover (Snargate)
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep 002320204 > /home/gm4slv/dover_formal_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep 002320204 > /home/gm4slv/dover_formal_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep Snargate > /home/gm4slv/dover_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep Snargate > /home/gm4slv/dover_8mhz.txt

#look for distress
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep distress > /home/gm4slv/distress_formal_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep distress > /home/gm4slv/distress_formal_2mhz.txt
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep distress > /home/gm4slv/distress_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep distress > /home/gm4slv/distress_8mhz.txt

# Look for "not Test"
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep -v test > /home/gm4slv/nottest_formal_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep -v test > /home/gm4slv/nottest_formal_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep -v test > /home/gm4slv/nottest_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep -v test > /home/gm4slv/nottest_8mhz.txt

# Look for Shetland
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep Shetland > /home/gm4slv/shetland_2mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep 002320001 > /home/gm4slv/shetland_formal_2mhz.txt

# Look for UK
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep CG]-G > /home/gm4slv/ukcg_2mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep 0023200 > /home/gm4slv/ukcg_formal_2mhz.txt

# Look for USA
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep ]-USA > /home/gm4slv/usacg_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep ]-USA > /home/gm4slv/usacg_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep 00366 > /home/gm4slv/usacg_formal_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep 00366 > /home/gm4slv/usacg_formal_8mhz.txt

# Look for Coast Stations
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep 'from;\[' > /home/gm4slv/coast_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep 'from;\[' > /home/gm4slv/coast_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep 'from;00' > /home/gm4slv/coast_formal_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep 'from;00' > /home/gm4slv/coast_formal_8mhz.txt

# Look for Ship Stations
cat /home/gm4slv/gm4slv_2mhz_dsc.txt | grep -v 'from;\[' > /home/gm4slv/ship_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_dsc.txt | grep -v 'from;\[' > /home/gm4slv/ship_8mhz.txt
cat /home/gm4slv/gm4slv_2mhz_formaldsc.txt | grep -v 'from;00' > /home/gm4slv/ship_formal_2mhz.txt
cat /home/gm4slv/gm4slv_8mhz_formaldsc.txt | grep -v 'from;00' > /home/gm4slv/ship_formal_8mhz.txt


# Remove blank lines on grep -v'd files
sed -i '/^$/d' /home/gm4slv/nottest_2mhz.txt
sed -i '/^$/d' /home/gm4slv/nottest_8mhz.txt
sed -i '/^$/d' /home/gm4slv/nottest_formal_2mhz.txt
sed -i '/^$/d' /home/gm4slv/nottest_formal_8mhz.txt
sed -i '/^$/d' /home/gm4slv/ship_2mhz.txt
sed -i '/^$/d' /home/gm4slv/ship_8mhz.txt
sed -i '/^$/d' /home/gm4slv/ship_formal_2mhz.txt
sed -i '/^$/d' /home/gm4slv/ship_formal_8mhz.txt

# Put new-lines on the filtered files
sed -i 's/$/\n/g' /home/gm4slv/dover_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/dover_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/dover_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/dover_formal_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/shetland_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/shetland_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/distress_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/distress_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/nottest_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/nottest_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/nottest_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/nottest_formal_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/distress_formal_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/distress_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ukcg_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ukcg_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/usacg_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/usacg_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/usacg_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/usacg_formal_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/coast_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/coast_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/coast_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/coast_formal_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ship_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ship_8mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ship_formal_2mhz.txt
sed -i 's/$/\n/g' /home/gm4slv/ship_formal_8mhz.txt

# Remove the ; on the Decoded files
sed 's/;/ /g' /home/gm4slv/gm4slv_2mhz_dsc.txt > /home/gm4slv/Dropbox/Public/gm4slv_2mhz_dsc.txt
sed 's/;/ /g' /home/gm4slv/gm4slv_8mhz_dsc.txt > /home/gm4slv/Dropbox/Public/gm4slv_8mhz_dsc.txt
sed 's/;/ /g' /home/gm4slv/shetland_2mhz.txt > /home/gm4slv/Dropbox/Public/shetland_2mhz.txt
sed 's/;/ /g' /home/gm4slv/distress_2mhz.txt > /home/gm4slv/Dropbox/Public/distress_2mhz.txt
sed 's/;/ /g' /home/gm4slv/distress_8mhz.txt > /home/gm4slv/Dropbox/Public/distress_8mhz.txt
sed 's/;/ /g' /home/gm4slv/nottest_2mhz.txt > /home/gm4slv/Dropbox/Public/nottest_2mhz.txt
sed 's/;/ /g' /home/gm4slv/nottest_8mhz.txt > /home/gm4slv/Dropbox/Public/nottest_8mhz.txt
sed 's/;/ /g' /home/gm4slv/ukcg_2mhz.txt > /home/gm4slv/Dropbox/Public/ukcg_2mhz.txt
sed 's/;/ /g' /home/gm4slv/usacg_2mhz.txt > /home/gm4slv/Dropbox/Public/usacg_2mhz.txt
sed 's/;/ /g' /home/gm4slv/usacg_8mhz.txt > /home/gm4slv/Dropbox/Public/usacg_8mhz.txt
sed 's/;/ /g' /home/gm4slv/dover_2mhz.txt > /home/gm4slv/Dropbox/Public/dover_2mhz.txt
sed 's/;/ /g' /home/gm4slv/dover_8mhz.txt > /home/gm4slv/Dropbox/Public/dover_8mhz.txt
sed 's/;/ /g' /home/gm4slv/coast_2mhz.txt > /home/gm4slv/Dropbox/Public/coast_2mhz.txt
sed 's/;/ /g' /home/gm4slv/coast_8mhz.txt > /home/gm4slv/Dropbox/Public/coast_8mhz.txt
sed 's/;/ /g' /home/gm4slv/ship_2mhz.txt > /home/gm4slv/Dropbox/Public/ship_2mhz.txt
sed 's/;/ /g' /home/gm4slv/ship_8mhz.txt > /home/gm4slv/Dropbox/Public/ship_8mhz.txt

# Add Tabs on the Formal Files
sed 's/;/\t/g' /home/gm4slv/gm4slv_2mhz_formaldsc.txt >/home/gm4slv/Dropbox/Public/gm4slv_2mhz_formaldsc.txt
sed 's/;/\t/g' /home/gm4slv/gm4slv_8mhz_formaldsc.txt >/home/gm4slv/Dropbox/Public/gm4slv_8mhz_formaldsc.txt
sed 's/;/\t/g' /home/gm4slv/nottest_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/nottest_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/nottest_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/nottest_formal_8mhz.txt
sed 's/;/\t/g' /home/gm4slv/distress_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/distress_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/distress_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/distress_formal_8mhz.txt
sed 's/;/\t/g' /home/gm4slv/dover_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/dover_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/shetland_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/shetland_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/dover_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/dover_formal_8mhz.txt
sed 's/;/\t/g' /home/gm4slv/ukcg_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/ukcg_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/usacg_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/usacg_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/usacg_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/usacg_formal_8mhz.txt
sed 's/;/\t/g' /home/gm4slv/coast_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/coast_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/coast_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/coast_formal_8mhz.txt
sed 's/;/\t/g' /home/gm4slv/ship_formal_2mhz.txt > /home/gm4slv/Dropbox/Public/ship_formal_2mhz.txt
sed 's/;/\t/g' /home/gm4slv/ship_formal_8mhz.txt > /home/gm4slv/Dropbox/Public/ship_formal_8mhz.txt


2 comments:

  1. It seems as the logs at dropbox is not working?
    M-A-

    ReplyDelete
  2. Yes, sorry. I'm testing a new decoder which is being developed for DSC, called YaDD (Yet Another DSC Decoder) and my previous method of splitting up DSCDecoder's logfiles is no longer appropriate. Perhaps YaDD will one day be able to send its received messages out in a suitable format for doing something similar, but until then, no more online DSC logs.

    ReplyDelete