NTP users are strongly urged to take immediate action to ensure that their NTP daemons are not susceptible to being used in distributed denial-of-service (DDoS) attacks. Please also take this opportunity to defeat denial-of-service attacks by implementing Ingress and Egress filtering through BCP38.

ntp-4.2.8p18 was released on 25 May 2024 and addresses 40 bugs and provides 40 improvements.

Please see the NTP 4.2.8p18 Changelog for details.

Bug 2901 - Clients that receive a KoD should validate the origin timestamp field.
Summary: Clients that receive a KoD should validate the origin timestamp field.
Status: RESOLVED FIXED
Alias: None
Product: ntp
Classification: Unclassified
Component: ntpd (show other bugs)
Version: 4.2.8
Hardware: All All
: P3 blocker
Assignee: Harlan Stenn
URL:
Depends on:
Blocks: 2952
  Show dependency tree
 
Reported: 2015-08-24 20:20 UTC by Majdi S. Abbas
Modified: 2024-07-23 18:21 UTC (History)
9 users (show)

See Also:
stenn: blocking4.2.8+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Majdi S. Abbas 2015-08-24 20:20:28 UTC
4.2.6p5 and 4.2.8p2/3 clients have been observed to honor any received KoD with the source address of a sys.peer, even when the packet contains an origin timestamp that does not match the one in the client's chime request.
Comment 1 Danny Mayer 2015-08-25 01:47:47 UTC
No, any KOD packet received should be dropped. The content of the timestamps are of no value in such a case. See RFC 5905 Section 7.4. If there is a problem it is that it may not be dropping the packet. Is that what you are observing?
Comment 2 Danny Mayer 2015-08-25 01:49:22 UTC
I also don't think this is a security issue.
Comment 3 Majdi S. Abbas 2015-08-26 00:22:09 UTC
(In reply to comment #1)
> No, any KOD packet received should be dropped. The content of the timestamps
> are of no value in such a case. See RFC 5905 Section 7.4. If there is a problem
> it is that it may not be dropping the packet. Is that what you are observing?

The observations aren't mine, but the problem is that the implementation is honoring, in client mode, KoDs it receives even if they don't match the chime request.  Validating that the KoD matches the original request helps to mitigate a possible attack where an attacker spoofs KoDs from a clients upstream servers, and by walking the refIDs (at least for v4), successfully denies time service to that client (or worse, steers it towards one hostile upstream server.)

> I also don't think this is a security issue.

Unfortunately in a world where BCP38 compliance is far from universal, it's possible to knock a client offline using only a few spoofed packets.  

Validating the origin timestamp returned to us matches the one sent is low cost and would help to up the bar on these sorts of attacks.
Comment 4 Harlan Stenn 2015-08-26 05:10:20 UTC
There are clearly problems with any choice here.

Majdi is right, it's possible to launch a DOS attack against a server by sending it a properly-chosen set of packets designed to throw off the clock and then send KoD packets to prevent the target from acquiring correct time from good servers.

I'm now wondering about the situation where if the association is authenticated, we'd only believe a KoD packet from the server if it was similarly authenticated.

There are potential problems no matter what we do here.

There are conflicting/complementary mechanism choices here.  We need to let folks make local policy choices.
Comment 5 Danny Mayer 2015-08-28 03:18:31 UTC
(In reply to comment #4)
> There are clearly problems with any choice here.
> 
> Majdi is right, it's possible to launch a DOS attack against a server by
> sending it a properly-chosen set of packets designed to throw off the clock and
> then send KoD packets to prevent the target from acquiring correct time from
> good servers.
> 

Well not really. If you are properly set up you get multiple packets from different servers and it's extremely hard for an attacker to do this to all of them. In any case, for DENY or RSTR codes the recipient of a kiss code packet is required to drop the association. For RATE it needs to reduce the frequency of queries. 

For ALL cases the recipient is required to drop the packet. The values of the timestamps cannot be used. I've had discussions with Dave Mills on some of this and it is clear that this is the design of the kiss codes. Under NO circumstances can the contents of the packets be used so even if there was a DOS attack these packets could not be used for anything to set the clock.

The biggest worry has been the clients that abuse the upstream NTP servers (this is especially a problem with government servers) and is far more important and required than possible attack vectors.


> I'm now wondering about the situation where if the association is
> authenticated, we'd only believe a KoD packet from the server if it was
> similarly authenticated.
> 

It doesn't matter, the packet needs to be dropped. 

> There are potential problems no matter what we do here.
> 
> There are conflicting/complementary mechanism choices here.  We need to let
> folks make local policy choices.

The problem raised is more theoretical than real. Any change would be detrimental to real live and busy systems.
Comment 6 Majdi S. Abbas 2015-08-28 05:19:28 UTC
(In reply to comment #5)
> Well not really. If you are properly set up you get multiple packets from
> different servers and it's extremely hard for an attacker to do this to all of
> them. In any case, for DENY or RSTR codes the recipient of a kiss code packet
> is required to drop the association. For RATE it needs to reduce the frequency
> of queries.

        It turns out it's not that hard, at least in an IPv4 world where
we helpfully place the IP address of the server in the refid field.
For v6, provided the server is not publicly listed anywhere the
truncated hash protects it to some degree.  For public servers it's not hard
to pre-compute a table of hash entries, limiting the number of possible
v6 addresses to source an attack from.

> For ALL cases the recipient is required to drop the packet. The values of the
> timestamps cannot be used. I've had discussions with Dave Mills on some of
> this and it is clear that this is the design of the kiss codes. Under NO
> circumstances can the contents of the packets be used so even if there was a
> DOS attack these packets could not be used for anything to set the clock.

        This has nothing to do with setting the clock; it has to do with
validating that a KoD received actually came from our associated server,
and wasn't spoofed.  Since the origin TS comes from the client, using it
is not providing time service -- it just gives the client a way to 
validate bidirectional communication with the server.

> The biggest worry has been the clients that abuse the upstream NTP servers
> (this is especially a problem with government servers) and is far more
> important and required than possible attack vectors.

        At this point the biggest worry is going to be selective denial
of service attacks by spoofing chimes from a client towards their 
associated servers to trigger rate limits, thus knocking them offline.  
I have no answer there other than overprovisioning and generous rate 
limits.

> It doesn't matter, the packet needs to be dropped.

        The packet does not need to be dropped; the association needs to
be dropped, or at least rate limited, provided that the origin timestamp is 
the one the client sent, which helps to validate that it came from the server (or at least an attacker with access to the network path.)

        Processing one timestamp out of the packet (which was originally
provided by the client to begin with), on the client itself, just to
validate the KoD, seems fairly safe.

> The problem raised is more theoretical than real. Any change would be
> detrimental to real live and busy systems.

        Unfortunately NTF is facing a series of soon-to-be-published
attacks around the KoD functionality.  Validating the origin TS on
the client may be the only way to save KoD in campus environments (it's
probably dead on the wider Internet after this publication.)

        Since the origin TS is copied to the transmit and receive TS
fields when sending a KoD, this involves no change to the KoD generation
code, so it should not affect deployed systems or harm anything in the field.

        It just allows us to slightly harden up the clients handling of a received KoD by validating bidirectional communication.
Comment 7 Danny Mayer 2015-08-28 20:11:56 UTC
I've received updated information on this and will be reviewing this.
Comment 8 Danny Mayer 2015-09-27 02:36:11 UTC
I have created a repository here ~mayer/ntp-4.2.8-Bug2901 that makes changes to drop any bogus packets. Only if the packet is validated will it check for KoD's and for RATE it will adjust the rate and it will drop all KoD packets.
Comment 9 Danny Mayer 2015-09-27 02:38:07 UTC
Ready for review.
Comment 10 Harlan Stenn 2015-10-06 08:26:18 UTC
BU folks, thanks for the report.

Danny, thanks for your work on this.

STAGED for 4.2.8p4.
Comment 11 Harlan Stenn 2015-10-23 06:27:56 UTC
Fixed in 4.2.8p4 and 4.3.77
Comment 12 Miroslav Lichvar 2016-01-21 09:25:35 UTC
As discussed on the ntp hackers list couple months ago, this issue doesn't seem to be fully fixed. When the TEST3 check performed on the transmit timestamp fails, the TEST2 check is skipped and a KoD reply is accepted even when its origin timestamp doesn't match the transmit timestamp from the request.
Comment 13 Harlan Stenn 2016-01-21 11:41:27 UTC
Miroslav,

I'm marking this as IN_PROGRESS per your comment.

I don't yet see what you are describing.

In ntp_proto.c, starting at line 1462, we have the case where:

- we are not in interleave mode
- if we are not a KOD packet {(in this case we *are*) clear the origin stamp}
- else if(aorg is zero, or the org stamps differ) we:
- - flash TEST2
- - log a message about unexpected origin timestamps
- - see if this qualifies for starting interleave
- - RETURN

This case should cover this bug report - a KoD packet is tested for origin timestamp.  Only if the origin timestamps are OK do we get past this point.

- else we clear the origin timestamp

Otherwise, we *are* handling interleave.

We look for a TEST3 violation.

If there isn't a TEST3 violation we do TEST2 checks for interleaved symmetric mode.

I think I may see a case here - the case where we get a KoD response and we're in interleave mode.

Or do you see something else I'm missing?

Now I'm curious about exactly what should happen with KoD packets in interleave mode.
Comment 14 Miroslav Lichvar 2016-01-22 08:02:59 UTC
In the current stable code on line 1416 in ntp_proto.c is checked the transmit timestamp. When it's zero the code that checks the origin timestamp starting at line 1462 is not executed. A packet that has zero transmit timestamp avoids validation of the origin timestamp.
Comment 15 Harlan Stenn 2016-01-23 13:56:52 UTC
I've refactored the way KoD packets are checked to include more than just the origin timestamp (the original thrust of this report).

These fixes are STAGED for 4.2.8p7.
Comment 16 Harlan Stenn 2016-04-27 04:04:20 UTC
Majdi,

Thanks for the report.  Please check 4.2.8p7 and mark this bug as VERIFIED or IN_PROGRESS, as appropriate.
Comment 17 Ulrich Windl 2016-06-29 10:38:03 UTC
In 4.2.8p8 I got this message (which is not very helpful):
28 Jun 16:04:25 ntpd[17516]: receive: KoD packet from 172.20.16.7 has inconsistent xmt/org/rec timestamps.  Ignoring.

Maybe more details should be added, like in this message:
28 Jun 16:00:04 ntpd[17516]: receive: Unexpected origin timestamp 0xdb1d0120.83568ef6 does not match aorg 0000000000.00000000 from sym_active@172.20.16.3 xmt 0xdb1d0164.82428466