The "Kiss-o'-Death"

SandyB · March 23, 2019, 4:16pm

Hi Bob,
I just noticed a few slightly worrying messages in the log:
3/20/19 13:16:27 timesync: Kiss-o’-Death, code RATE, ip: 45.76.244.193
3/20/19 13:16:36 timesync: Kiss-o’-Death, code RATE, ip: 45.76.244.193
3/20/19 13:16:46 timesync: Kiss-o’-Death, code RATE, ip: 45.76.244.193
3/20/19 13:16:55 timesync: Kiss-o’-Death, code RATE, ip: 45.76.244.193
Though everything seems to be working fine anyway
ALPHA channel, running 02_04_00. Up since 3/12/19, and no more messages since 3/20/19
Thanks,
Sandy

overeasy · March 23, 2019, 6:09pm

Yea, that’s nothing to worry about.

IoTaWatt checks the time with pool.ntp.org every hour. The pool servers, in an effort to minimize load, push back when they get a lot of requests from a specific IP. This is their way of telling IoTaWatt to back off, which it does… somewhat.

I have tried to register IoTaWatt as a regular user of the service, but the request has gone unanswered. So beginning with the next release, I will be switching over to time.google.com. For all the capacity problems that ntp.org has, this is a trivial service for google, who are in a better position to do this globally anyway.

Agnes · January 1, 2022, 7:22pm

Sometimes my NTP server returns a KOD packet to the IoTaWatt.
12/30/21 17:54:11 timesync: Kiss-o'-Death, code INIT, ip: 192.168.1.1

I run the latest version of NTP server (4.2.8p15) on my local network, backed by a GPS reference clock up on the roof. I also redirect any local NTP client requests to my NTP server, and any local DNS lookups of known internet time servers to my NTP server. When the IoTaWatt tries to synchronize its clock with time.google.com, my router redirects it to 192.168.1.1. It seems to work fine, however it may cause the NTP clients to make more than one timesync request, due to my DNS redirect?

If I could turn this rate-limiting off I would, since my NTP server is only accessible from the local network. I found issues with the NTP documentation for the “limit” and “unrestrict” commands.
https://docs.ntpsec.org/latest/ntp_conf.html#_access_control_commands

Following is my NTP server configuration. Maybe there’s a switch to disable those KOD packets or soften the rate-limiting, in the NTP server configuration?

#
# /etc/ntp.conf
#
driftfile /tmp/etc/ntp/ntp.drift
leapfile /jffs/ntp/leap-seconds.list
statsdir /var/spool/ntp
keysdir /jffs/ntp
pidfile /var/run/ntpd.pid
logconfig=allall
interface listen lo
interface listen br0
restrict default nomodify notrap nopeer noquery
restrict -6 default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1

# Generic NMEA GPS Receiver
server 127.127.20.0 mode 24 prefer true # select bitrate, $GPZDA, truechimer
fudge 127.127.20.0 stratum 0 # the driver stratum, in decimal from 0 to 15, with default 0
fudge 127.127.20.0 refid PPS # an ASCII string from one to four characters, with default GPS
fudge 127.127.20.0 flag1 1 # Enable PPS signal processing
fudge 127.127.20.0 flag2 0 # Capture the pulse on the rising edge
fudge 127.127.20.0 flag3 1 # Use the kernel discipline
fudge 127.127.20.0 flag4 1 # Obscures location in timecode

# Undisciplined Local Clock
server 127.127.1.0
fudge 127.127.1.0 stratum 5 # the driver stratum, in decimal from 0 to 15, with default 5
fudge 127.127.1.0 refid LOCL # an ASCII string from one to four characters, with default LOCL

overeasy · January 1, 2022, 8:21pm

This is the main reason I got off of. NTP servers and went with time.google. NTP not only does rate limiting, but it’s different with every server. IoTaWatt is installed in at least 60 countries worldwide and each one can be configured differently. As I understand it the NTP.org system is a volunteer organization. I tried to register IoTaWatt as a global user in the hope it would improve, but the administrators are volunteers as well and never replied to multiple requests to register.

For google’s global server network serving time is child’s play and has been rock solid since I switched over. So the protocol is the same and you are free to redirect to the NTP server software, but the problems come back in the bargain.

Agnes · January 2, 2022, 5:39pm

Just noticed that the Reference-Id of my KOD reply was “INIT”, not “RATE”. This means it’s not a real KOD. And I verified that my NTP server is not doing rate limiting or sending KOD packets. I think the issue is that my Stratum number is zero, which means that I’m a root time server (i.e. an NTP server with a reference clock). After reviewing the IoTaWatt source code, I could easily change my Stratum number to 1 in /etc/ntp.conf, which would resolve my specific issue.

github.com

boblemaire/IoTaWatt/blob/4b021e3657d11ef54e6a313fc0d3581ec5958ccf/Firmware/IotaWatt/timeServices.cpp#L147


      
                // Validate packet.
          
          
trace(T_timeSync, 6);
          if(packetSize < sizeof(ntpPacket) || recvMillis - sendMillis > (RTCrunning ? 3000 : 10000)){
            return UTCtime() + (RTCrunning ? 60 : 5);
          }
          
          
      // Check for Kiss-o'-Death packet
          
          
trace(T_timeSync, 7);
          if(packet.stratum == 0){
            log("timesync: Kiss-o'-Death, code %c%c%c%c, ip: %s", 
            packet.referenceID[0], packet.referenceID[1], packet.referenceID[2], packet.referenceID[3], timeServerIP.toString().c_str());
            return UTCtime() + (RTCrunning ? 60 : 15);
          } 
          
          
trace(T_timeSync, 8);
          if(packet.origin_ts_sec != origin_sec || packet.origin_ts_frac != origin_frac){
            trace(T_timeSync, 81);
            return UTCtime() + (RTCrunning ? 60 : 5);
          }

Agnes · January 2, 2022, 6:14pm

On second thought, I don’t think changing my Stratum number will fix this. So, I will leave tcpdump running on my router until it happens again. Then we can view the captured packets in Wireshark.

FYI, here’s how SNTP checks for KOD. It seems there’s more to this than simply checking if Packet.Stratum==0.

github.com

ntp-project/ntp/blob/9c75327c3796ff59ac648478cd4da8b205bceb77/sntp/networking.c#L237


      
          		return SERVER_UNUSEABLE;
          	} 
          	/* We want a server to sync with */
          	if (PKT_MODE(rpkt->li_vn_mode) != mode &&
          	    PKT_MODE(rpkt->li_vn_mode) != MODE_PASSIVE) {
          		msyslog(LOG_ERR,
          			"%s: mode %d stratum %d", func_name, 
          			PKT_MODE(rpkt->li_vn_mode), rpkt->stratum);
          		return SERVER_UNUSEABLE;
          	}
          	/* Stratum is unspecified (0) check what's going on */
          	if (STRATUM_PKT_UNSPEC == rpkt->stratum) {
          		char *ref_char;
          
          
		TRACE(1, ("%s: Stratum unspecified, going to check for KOD (stratum: %d)\n", 
          			  func_name, rpkt->stratum));
          		ref_char = (char *) &rpkt->refid;
          		TRACE(1, ("%s: Packet refid: %c%c%c%c\n", func_name,
          			  ref_char[0], ref_char[1], ref_char[2], ref_char[3]));
          		/* If it's a KOD packet we'll just use the KOD information */
          		if (ref_char[0] != 'X') {

Agnes · January 2, 2022, 7:33pm

So I just discovered this happens when my NTP server has no time sources available, the IoTaWatt logs a KoD with a ref id of INIT instead of RATE. Since the IoTaWatt is expecting to synchronize with time1.google.com, it should never be the case that Google is down.

You might be able to detect this situation, and not log a KoD INIT?

#define	MODE_SERVER	4
#define	LEAP_NOTINSYNC	0x3
if( (packet.flags & 0x7 == MODE_SERVER) &&
    (packet.flags >> 6) & 0x3 == LEAP_NOTINSYNC) )
{
  // the time server is not ready yet, please try again later
}

overeasy · January 2, 2022, 8:10pm

Since changing to time.Google.com there have been zero problems around the world. Making any kind of change without extensive testing risks the rule of unintended consequences. So without a compelling reason, or capability to test globally, I don’t think I’ll be making any changes.

Your options are to make these changes yourself to the open source firmware and maintain your own version, or possibly to set your NTP instance to simply not respond if it doesn’t know what time it is.

It’s also not clear that this is a serious problem as IoTaWatt will continue to retry until it succeeds.

Agnes · January 2, 2022, 11:43pm

I think it’s more important to get automatic updates than run a custom version with my own fixes. If anyone else runs into this issue they can search this forum and see that a KoD INIT is completely different than a KoD RATE (as reported by the OP).

Lastly, during troubleshooting I saw strange characters in the log for the Reference-Id. These were all caused by an NTP server without a valid time source.

1/02/22 13:52:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:53:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:54:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:55:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:56:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:57:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:58:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 13:59:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 14:00:35 timesync: Kiss-o'-Death, code ��, ip: 192.168.1.1
1/02/22 14:01:35 timesync: Kiss-o'-Death, code INIT, ip: 192.168.1.1

DerekD · January 3, 2022, 4:08am

I am new to the IoToWatt community. )

Has there been any thoughts about a configuration screen option to set a time server of choice by the admin and/or accept/use DHCP option 42 (RFC 2132 section 8.3)? Having a default in the firmware such as Google would be OK.

The Internet has changed a lot, but hard coding something in firmware reminds me of Netgear and the flood of traffic to University of Wisconsin’s NTP servers 19 years ago. See Flawed Routers Flood University of Wisconsin Internet Time Server for a nice write up from 2003. With the way Google anycasts many of their network services I don’t think the deployed IoTaWatts will ever cause a problem.

iotapl · January 3, 2022, 4:31pm

I’d second this. Having a way to configure the NPT to another service. I have an NTP server at home and I try to get every device talk to that. The idea is that the least amount of traffic in/out of my IoT network the better for security.

donavan · October 6, 2022, 7:28pm

Agree with the above. IoTaWatt should at least honor the NTP server sent with the DHCP response.

If not NTP should be user configurable without maintaining a custom firmware. Not everyone is capable of intercepting requests and redirecting them to a local time server.

In addition, the updates should be done the same way. Let me pick the server where my IoTaWatts look for updates.

Hide the options behind an advanced tab where the user commits to self support if they change settings.