This document provides details of a DHCP bug exhibited by Mac OS X 10.5.x in rare circumstances. We have confirmed its presence in 10.5.6 and 10.5.8. We suspect it is present in other versions of Mac OS X 10.5.x.
The bug can cause the Mac to interfere with service to other devices on the network. This document provides information for individuals who would like detailed technical information about the issue.
Princeton University reported the bug to Apple. This bug was fixed in Mac OS X 10.6.
DHCPv4, the Dynamic Host Configuration Protocol for IPv4, allows a device attached to the network to automatically learn some or all of its network configuration, including its IPv4 (Internet) address. Most operating systems include DHCP client software.
A device that uses DHCPv4 runs DHCP client software on a network (e.g., Ethernet or Wireless) interface. The DHCP client software contacts DHCP servers to obtain network configuration; in particular, it usually obtains a lease (a loan) of an IP address.
For example, the DHCP client tells the DHCP server "I am network interface 0:1:2:3:4:5; please lease an IP address to me." The DHCP server might respond "You may use IP address 192.168.1.2 for the next six hours; if you would like to continue using that address, please renew it when three hours have elapsed." When three hours have elapsed, the DHCP client contacts the DHCP server which granted the lease; the client asks that server to renew the lease Typically the DHCP server responds to the client: "You may use IP address 192.168.1.2 for the next six hours; if you would like to continue using that address, please renew it again when three hours have elapsed." (If the DHCP client is unable to contact the DHCP server to the renew its unexpired lease, it will retry from time to time, and is permitted to continue using the IP address until the lease is due to expire.)
Assuming the DHCP client successfully renews the lease before it expires, this repeats periodically until the device goes offline. Once the device is offline, it no longer contacts the DHCP server to renew the lease, so eventually the last lease renewal expires. Once the last lease renewal has expired, the DHCP server is free to lease the IP address to another client.
If the device goes offline, when it later comes back online it broadcasts a DHCP request for a new lease. It may choose to request a brand-new lease, or (if it believes the old lease has not yet expired) may request a new lease on the old IP address.
A DHCP client which wishes to surrender a leased IP address before the lease expires may optionally contact the DHCP server to release the lease on that IP address. The DHCP server may then lease that IP address to another client.
Devices running Mac OS X 10.5.x sometimes get into a state in which they release a DHCP lease, but continue to use the IP address from that lease.
The pattern we see is that the device obtains a lease on IP address 'A'. Prior the expiration of the lease, the device sends a DHCPRELEASE message to the DHCP server, releasing the lease on IP address 'A'. Later, the device uses DHCP to obtain a lease on IP address 'B'. It proceeds to use IP address 'B' and also IP address 'A' at the same time. That is, it uses its new legitimate IP address 'B', but simultaneously "steals" IP address 'A'.
Later it may use DHCP to obtain a lease on IP address 'C'. It proceeds to use IP address 'C' and also IP address 'A' at the same time. That is, it uses its legitimately-leased IP address, but simultaneously "steals" IP address 'A' which it released hours or days ago.
This continues on and off until the problem with the device is addressed through manual intervention (described below).
If the problem is not addressed, eventually the problem worsens. The device exhibits the problem a second time, releasing a lease on IP address 'C'. From that time on, the device will steal both IP addresses 'A' and 'C' at the same time as it uses any other legitimately-obtained IP address 'D'. Over time, the device accumulates old IP addresses it has released, stealing all of them in addition to using its current legitimate IP address.
This is interferes with service to other devices on the network, as after the device releases an IP address, the DHCP server is free to lease the IP address to another device.
We do not know what causes the device to enter this malfunctioning state.
Apparently the circumstances leading a device to enter this malfunctioning state are not common, as we have seen only several dozen (out of thousands) exhibit the problem. We have observed that some of the devices which have exhibited this problem have gotten into this malfunctioning state a number of times; we do not know what differentiates these devices from others.
A temporary workaround to the problem is to clear the networking configuration on the device (e.g., delete the configuration using the 'Network' System Preference pane). Afterwards, recreate its networking configuration anew. This ends the current incident, however, the underlying bug is still present, so the device may still malfunction in this way at some time in the future.
Princeton began seeing this malfunction in September 2009. As the frequency of the problem was low, it took a number of months before we recognized the pattern, and several more months to collect the necessary data to describe the bug.
In April 2010, we reported the bug to Apple.
Apple responded to our bug report; the non-disclosure agreement involving Apple bug reports prevents us from disclosing Apple's response, however, we can note that this bug was fixed in Mac OS X 10.6, released August 2010.
When Princeton encounters a device malfunctioning in this way, we temporarily block the device from our network to stop it from interfering with service. We contact the owner and ask him or her to take action (delete the device's network configuration) to address the current incident. We also advise the owner that the bug remains present in Mac OS X 10.5.x, and so is likely to recur in the future, until the owner uprades to Mac OS X 10.6. Once the customer indicates that the current incident has been addressed, we remove the temporary block.
Some may wonder why only Princeton has reported this problem. Some may believe that because other sites did not report it, the problem must have been due to a problem with Princeton's network.
Princeton detected this issue because we take a very pro-active stance to monitor for certain kinds of common network problems, including this one.
Our network monitoring includes comparing actual IP address usage to DHCP server lease assignments on a daily basis. Specifically, we compare our IP router ARP cache data to our DHCP server logs. This allows us to detect some devices using IP addresses not assigned for their use. This is a degree of monitoring that many sites do not perform. As many sites place client devices -- especially wireless clients -- behind NATs, performing such monitoring may be difficult for most sites. At the time we first encountered this Mac OS X issue, none of our client networks were behind NATs, making such monitoring somewhat easier for us.
We also monitor our DHCP servers very closely for any problems they detect, including when they see DHCP-leased IP addresses in-use when they should not be, or when a client tries to SELECT an offer that was not made to it, or when a client tries to renew or rebind an IP address after the client's lease on that IP address has already expired. We have instrumented our DHCP server software to make it (somewhat) easier to see such events. Our monitoring also reports DHCP clients which are the source of excessive transactions; occasionally these are victims of malfunctioning Mac OS X devices "stealing" IP addresses.
As a result of the close monitoring we perform to detect DHCP issues, Princeton tends to learn about some kinds of bugs in DHCP client implementations sooner and more often than do many other sites.
Our network monitoring includes comparing actual IP address usage to DHCP server lease assignments on a daily basis. This allows us to detect some devices using IP addresses not assigned for their use. This is a degree of monitoring that many sites do not perform. We also monitor our DHCP servers very closely for any problems they detect, including when they DHCP-leased IP addresses in-use when they should not be.As a result, Princeton tends to learn about some kinds of bugs in DHCP client implementations sooner and more often than do many other sites.
We could choose instead to not take a pro-active stance to these kinds of issues. A more common approach is to ignore the kinds of problems caused by devices using IP addresses not leased to them, allowing such malfunctioning devices to cause sporadic mysterious network problems for customers as their IP addresses are "stolen". Sites that use that approach may take action only when a victim of a malfunctioning device chooses to complain. Most victims probably don't complain because these kinds of problems appear random and short-lived to each victim, and often go away when they "try again."
We feel that the stance we take ultimately benefits our customers, as it results in more reliable network service to the customers. It reduces the frequency that our customers experience network disruptions due to others' malfunctioning devices.
As a side note, this pro-active stance has also resulted in our discovering DHCP client issues a number of times over the years for a variety of common platforms. Typically we've provided technical details of these issues to the DHCP client vendors, which has helped the vendors to fix bugs and improve DHCP client behavior. Although identifying issues in vendors' DHCP client software is not our goal -- our mission is to provide excellent network service to Princeton University customers -- it does speak to the technical accuracy of the bugs we've discovered.