NTP users are strongly urged to take immediate action to ensure that their NTP daemons are not susceptible to being used in distributed denial-of-service (DDoS) attacks. Please also take this opportunity to defeat denial-of-service attacks by implementing Ingress and Egress filtering through BCP38.

ntp-4.2.8p18 was released on 25 May 2024 and addresses 40 bugs and provides 40 improvements.

Please see the NTP 4.2.8p18 Changelog for details.

Bug 1331 - DoS with mode 7 packets (CVE-2009-3563)
Summary: DoS with mode 7 packets (CVE-2009-3563)
Status: RESOLVED FIXED
Alias: None
Product: ntp
Classification: Unclassified
Component: ntpd (show other bugs)
Version: (unspecified)
Hardware: PC All
: P3 normal
Assignee: Harlan Stenn
URL:
Depends on:
Blocks:
 
Reported: 2009-10-04 22:27 UTC by Danny Mayer
Modified: 2009-12-31 17:05 UTC (History)
3 users (show)

See Also:
stenn: blocking4.2.4+
stenn: blocking4.2.6+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Danny Mayer 2009-10-04 22:27:11 UTC
From dmitri vinokurov:
We believe there is a flaw in NTP which allows to build an effective and easy
exploitable DoS attack.
The topology used includes two nodes running ntp and an attacker's PC:

PC--->  [node1 ntpd1]:11.0.0.1 --------11.0.0.2:[node2 ntpd2]

PC sends one crafted UDP packet with one byte payload 0x17, i.e. NTP Request in
mode 7.
This UDP packet has spoofed source IP of 11.0.0.2, destination = 11.0.0.1,
source port 123 and destination port 123.
Node1 responds with mode 7 Error Response to Node2, and here comes something we
cannot conceive. Ntpd2 responds back with the same mode 7 Error Response to
Node1, Ntpd1 does again the same, etc. with the aggregate rate of few thousand
pps. CPU is taken away on both sides, network is busy...
Better yet, if we spoof the Node1's address 11.0.0.1 as a source, Node1 sends
all these packets to itself all the time! Endless.
Payload "97 00 00 00" (Response mode 7) works too.

We believe ntpd must either drop an unexpected mode 7 Response, or drop any mode
7 packets originated from port 123 (they have to use high number source ports).

Authentication and ACLs may help, but these are rather workarounds. Would it be
possible to address the core issue?
Comment 1 Danny Mayer 2009-10-04 22:34:25 UTC
I haven't looked at the code yet, but it should just drop any
packets that are shorter than the minimum size. That's the first
problem. The second problem is that a node receiving an error response
should not be responding automatically to it with it's own error
response. Receipt of an error response should not be responded to.

Danny
Comment 2 Danny Mayer 2009-10-04 22:39:42 UTC
From Dave Mills:

That code was last touched by Dennis Fergusson circa 1984. It should be fixed.

The cardinal rule, at least in the mainline code, is that packets with bad
format, wrong length or incorrect version are always dropped and do nothing
except increment a tattletale. Control/monitor packets can't be checked by the
mainline code, since the packet lengths are different.

As you know, I would much, much rather strip ntpdc of anything except raw debug.

There is actually a much easier way to mount a DoS attack. Thrwow monlist
commands at ntpdc at a nice fat rate. The monitoring tools should be subject to
the same rate controls as the mainline code.
Comment 3 Danny Mayer 2009-10-04 22:40:57 UTC
From dmitri vinokurov:

Dave, Danny,
Thanks for your comments and acknowledgement.
I believe there is still some misunderstanding though.
- the problem is NOT in lack of format/length/version control, sanity check,
etc. - all this can be made up in the first rogue packet. Danny got it right -
"Receipt of an error response should not be responded to", and this is the core
issue. Ntpd endlessly responds even to itself, on the same platform.
- the exploit is as trivial as to send one individual packet - ntpd does all the
flooding job for us by itself, and this is indeed easier than to throw whatever
commands at a fat rate. The result is - CPU load goes up to 100% on both sides
depending on the platform, network is busy, and no skills required to exploit
this. The overall risk is very high.
Comment 4 Danny Mayer 2009-10-05 04:47:36 UTC
I have checked in a fix in my repo on pogo ntp-dev-1331.

This repo is only for this fix.

Danny
Comment 5 Harlan Stenn 2009-10-05 04:59:32 UTC
Subject: DoS with mode 7 packets 

Thanks Danny.

I've asked CERT for a CVE for this issue.  I'll schedule the release
with them.

H



-- 
Harlan Stenn <stenn@ntp.org>
Comment 6 Dave Hart 2009-10-05 08:21:10 UTC
The fix in Danny's repo looks a bit wrong to me.  The test ISRESPONSE() is still 
being done by the code which will log a message and transmit a reply.  Shouldn't 
it be moved up to the new code which will return without logging or replying?
Comment 7 Harlan Stenn 2009-10-05 09:00:41 UTC
I'm presently going to let this bug block 4.2.6.
Comment 8 Danny Mayer 2009-10-05 12:48:06 UTC
Re comment #6. No, that deals with the version of the NTP packet and really does
not matter one way or another. We're ignoring responses in any case.

Danny
Comment 9 Dave Hart 2009-10-05 15:16:24 UTC
Danny, re comment #8, I'm confused.  In comment #3 you quote Dmitri correctly 
noting the crux of the issue is ntpd responding to mode 7 responses.  Yet your 
ntp-dev-1331 has this code:

  ec = 0;
  if (   (++ec, INFO_SEQ(inpkt->auth_seq) != 0)
      || (++ec, INFO_ERR(inpkt->err_nitems) != 0)
      || (++ec, INFO_MBZ(inpkt->mbz_itemsize) != 0)
      || (++ec, rbufp->recv_length < REQ_LEN_HDR)
          ) {
                  return;
  }

  if (   (++ec, ISRESPONSE(inpkt->rm_vn_mode))
      || (++ec, ISMORE(inpkt->rm_vn_mode))
      || (++ec, INFO_VERSION(inpkt->rm_vn_mode) > NTP_VERSION)
      || (++ec, INFO_VERSION(inpkt->rm_vn_mode) < NTP_OLDVERSION)
          ) {
          msyslog(LOG_ERR, "process_private: INFO_ERR_FMT: test %d failed, pkt 
from %s", ec, stoa(srcadr));
          req_ack(srcadr, inter, inpkt, INFO_ERR_FMT);
          return;
  }

This fixes ntpd responding to runt packets and those with nonzero auth_seq, 
err_nitems, and mbz_itemsize, but it leaves the crux of the issue in place, by 
still logging and replying to responses.  I'm not talking about the version 
checks in the second if, I'm talking about the ISRESPONSE() test.  Why should 
ntpd log and reply to a response?  I would reopen the bug, but I'm not 
interested in playing bugzilla ping-pong.
Comment 10 Danny Mayer 2009-10-05 16:13:33 UTC
I have now fixed that in the repository. I managed to miss that. It's probably
not going to make much of a difference but the possibility is something to be
managed.

Danny
Comment 11 Harlan Stenn 2009-10-06 00:19:04 UTC
Updated to include CVE-2009-3563 .
Comment 12 Harlan Stenn 2009-10-06 00:19:51 UTC
Dmitri,

What "citation" should we use when crediting the "discovery" of this CVE?
Comment 13 Dmitri Vinokurov 2009-10-06 03:35:44 UTC
Subject: DoS with mode 7 packets (CVE-2009-3563)

Harlan,
To be honest, I am struggling with the translation of the term 
"citation" in this context.
My best bet would be the name(s) and affiliation of those that the 
credits go to. Is that right?
If so, you can credit Robin Park an myself of Alcatel-Lucent.
Thank you,
Dmitri

Harlan Stenn via the NTP Bugzilla wrote:
[---=| TOFU protection by t-prot: 11 lines snipped |=---]



-- 
Dmitri Vinokurov <dmitri.vinokurov@alcatel-lucent.com>
Comment 14 Dave Hart 2009-10-07 01:59:04 UTC
After studying how ntp_control.c handles similar issues with mode 6 packets, I 
have a slightly different fix prepared.  My patch leaves the original big if ( 
ec++, ... block intact, and changes its body in two ways.  First, for any of 
those initial sanity checks, a failure will not elicit a response.  This is the 
way mode 6 and mainline code deals with malformed packets.  Second, the msyslog 
is made conditional on NLOG_SYSEVENT and rate-limited to once per minute.  The 
rate limiting allows ntpd to log the fact of malformed or malicious packets in 
this path without risking flooding if, for example, a broadcast or multicast 
address ntpd is listening on is targeted.  In that situation, the fact that ntpd 
no longer responds may not be enough to stop the flood, as others sharing the 
address may be responding.

This is prepared as a ntp-stable delta pulled into ntp-dev in:

psp-deb1:~hart/ntp-stable-1331
psp-deb1:~hart/ntp-dev-1331

As it touches only ChangeLog and ntpd/ntp_request.c it would be trivial to make 
an equivalent freestanding ntp-dev-1331 if the decision is made to fix it in 
ntp-dev alone first.
Comment 15 Danny Mayer 2009-10-07 03:13:54 UTC
The number one rule of security fixes is to keep the fixes simple so that
vendors, if necessary can backport the fix to earlier releases. There are
assumptions here in your proposed changes that are only valid for the latest
releases and should not be assumed to be valid for earlier releases. The
important part is to make clear what needs to be changed. This is not just for
4.2.4 but needs to be easy to do for earlier releases. Enhancements like you are
suggesting can happen later but should be done with additional thought to what
really needs to be logged and that has not been mapped out yet.

Danny
Comment 16 Dave Hart 2009-10-07 03:26:58 UTC
Danny, I agree it is important to keep the patch simple and focused.  If you 
compare my proposed patch with yours I think you'll see mine is simpler.  It 
does not split the early sanity checks into two parts, it does not rearrange the 
order of the tests, and thereby the meaning of the logged test numbers.

What it does do is keep to the ntpd practice in mode 6 and mainline processing 
of dropping malformed packets without a peep in responses, and add rate-limiting 
code to ensure the msyslog triggered by the big if statement happens no more 
than once per minute.

I stand by my proposed patch as a simpler, more focused fix and one that brings 
mode 7 handling in line with other packet input code paths in ntpd.

Please compare for yourself by browsing:

psp-deb1:~hart/ntp-dev-1331.pupatch.txt
and
psp-deb1:~hart/ntp-dev-1331-mayer.pupatch.txt
Comment 17 Danny Mayer 2009-10-07 03:58:25 UTC
One more check added to the dropped packet list and checked in.

Danny
Comment 18 Harlan Stenn 2009-10-22 06:05:09 UTC
I sent the patch to CERT and they are alerting the vendors.  We also have a VU
number from them.

CERT prefers to have at least 45 days' time for vendors to apply the patch and
test it.  They have asked us if Tuesday 8 December is an acceptable "announce"
date.  That seems fine with me, anybody disagree?
Comment 19 Danny Mayer 2009-10-28 02:55:38 UTC
Comment #14 raises an interesting point, that I haven't had a chance to check,
having overlooked it when I first read it. If a mode 6 or mode 7 packet arrives
from a multicast or broadcast address it should always be dropped even if the
packet is valid. In fact the only packets arriving on those addresses should be
the broadcast (mode 5) packets. I haven't looked to see if they are being
dropped. I'll open a separate bug report on this but I don't think it would
classify as a security issue.

Danny
Comment 20 Dave Hart 2009-10-28 03:18:06 UTC
(In reply to comment #19)
> If a mode 6 or mode 7 packet arrives
> from a multicast or broadcast address it should always be dropped even if the
> packet is valid.

Good idea.  Drop mode 6 and mode 7 packets addressed to a broadcast or multicast 
address (not from).  I would be very careful in testing all supported 
configurations before I'd attempt to drop mode 1-5 sent to *cast addresses.
Comment 21 Danny Mayer 2009-10-28 03:24:29 UTC
Yes, I meant to broadcast or multicast address. You meant dropping modes 1-4
since 5 is the broadcast packet.

Danny
Comment 22 Harlan Stenn 2009-11-04 21:03:56 UTC
Adding Brian Utterback to the Cc: list, as he received info from CERT about this
and it's easier to let him read this report than send him information.
Comment 23 Dave Hart 2009-11-10 18:26:29 UTC
Martin Burnicki and Heiko Gerstung of Meinberg should be notified so that they 
can prepare patched releases wrapped in updated installers to be ready for 
coordinated release.  Although uncompensated, Meinberg essentially is a NTP 
vendor in the CERT sense for Windows users.  In that regard, I regret not giving 
them 45 days like the rest of the vendors.  Even if there are no kinks building a 
patched release, I know it takes some time and steps to produce the installer.

They can be added to the CC list assuming there are no objections raised.
Comment 24 Dave Hart 2009-12-05 02:31:22 UTC
There are two restrict bits that if found will prevent reaching the buggy code.  
RES_IGNORE, and RES_NOQUERY.  So "restrict default noquery" and including noquery 
on all restrict lines except those where ntpq and ntpdc queries must be permitted 
is a workaround.
Comment 25 Danny Mayer 2009-12-06 05:12:16 UTC
This is already documented in the CERT. You cannot *any* permit any ntpq or
ntpdc queries if you want to be safe.

Danny
Comment 26 Harlan Stenn 2009-12-09 01:54:02 UTC
Danny,

There is no reason based on this ticket to restrict mode 6 packets.

The fix for this problem is in 4.2.4p8 and 4.2.6.
Comment 27 Danny Mayer 2009-12-09 02:09:38 UTC
Harlan, I'm not sure I understand your comment. If you are talking about comment
#25, you don't have a choice since the restrict statement does not allow you to
differentiate between mode 6 and mode 7, otherwise I would agree.

Danny
Comment 28 Brian Utterback 2009-12-29 19:11:57 UTC
Looking at the handling of the of the request packet, I am surprised that you
did not follow the handling in parallel with the way ntp_control handles control
packets. Or perhaps even simpler.

I believe that the following patch would also have eliminated the problem:

*** ntp_request.c.fcs   Tue Dec 29 13:57:14 2009
--- ntp_request.c       Tue Dec 29 13:59:34 2009
***************
*** 447,454 ****
         * error if it fails.
         */
        ec = 0;
!       if (   (++ec, ISRESPONSE(inpkt->rm_vn_mode))
!           || (++ec, ISMORE(inpkt->rm_vn_mode))
            || (++ec, INFO_VERSION(inpkt->rm_vn_mode) > NTP_VERSION)
            || (++ec, INFO_VERSION(inpkt->rm_vn_mode) < NTP_OLDVERSION)
            || (++ec, INFO_SEQ(inpkt->auth_seq) != 0)
--- 447,454 ----
         * error if it fails.
         */
        ec = 0;
!       if (rbufp->recv_length < REQ_LEN_HDR || (++ec,
ISRESPONSE(inpkt->rm_vn_mode))) return;
!       if (   (++ec, ISMORE(inpkt->rm_vn_mode))
            || (++ec, INFO_VERSION(inpkt->rm_vn_mode) > NTP_VERSION)
            || (++ec, INFO_VERSION(inpkt->rm_vn_mode) < NTP_OLDVERSION)
            || (++ec, INFO_SEQ(inpkt->auth_seq) != 0)


Of course, just as a matter of style, I think that the test for the data
actually being there should be done before looking at the data. I think that the
current set of tests will actually access uninitialized data in the order they
currently are in.

But the key field for this attack is the response bit. The req_ack function
always sets the response bit, so you can never get into a loop is you ignore
packets with the response bit set.  I don't see any reason to log it. If you
want to get fancy, then you could set up a counter. But the current fix is too
draconian, never sending back a response to any type of error and logging the
event in such a way as to require disable logging for as time. That seems kind
of silly.
Comment 29 Dave Hart 2009-12-31 17:05:09 UTC
(In reply to comment #28)
> Of course, just as a matter of style, I think that the test for the data
> actually being there should be done before looking at the data. I think that
> the current set of tests will actually access uninitialized data in the order
> they currently are in.

I'd agree if I were writing this code from scratch.  The patch was intended to 
be as narrow as possible while addressing the bug.  Moving the test for the 
received length first does not seem worth doing to me even in ntp-dev only, as 
the receive buffers are always big enough that we're not risking a fault by 
checking potentially uninitialized data, and if the leftover garbage being 
examined in processing a runt happens to pass all the other checks in that 
compound if with the "++ec"s, it will still be rejected due to the size.  So I'm 
failing to see what bug you'd be fixing by shuffling the length check to be 
first.

> But the key field for this attack is the response bit. The req_ack function
> always sets the response bit, so you can never get into a loop is you ignore
> packets with the response bit set.  I don't see any reason to log it. If you
> want to get fancy, then you could set up a counter.

The reason for the logging is so you can be aware someone is trying to involve 
your ntpd in a screaming match.  In particular, imagine the case where a 
broadcast configuration is targetted.  The logging would let you see which 
machines listening on the broadcast address are running vulnerable code and 
keeping the spew alive, so you can fix them.

> But the current fix is too
> draconian, never sending back a response to any type of error

Why is it important to send any response to a malformed packet?  Dropping 
without response is what is done for malformed packets in all the other ntpd 
paths.

> and logging the event in such a way as to require disable logging
> for as time. That seems kind of silly.

Logging is not disabled -- this one message is rate-limited to no more than once 
per minute, for those not using syslog with its duplicate suppression.  It could 
also help when a ntpd is involved in volleys with more than one other ntpd, 
where syslog duplicate suppression might not contain the log spew.

If you have a patch that you believe makes things less silly, by all means 
propose it.