OIT Networking & Monitoring Services

OIT Networking & Monitoring Services: Device Blocking Policies

NOTE: This document reflects policies through June 12 2018. OIT revised its device blocking policies June 13-15 2018. An updated document reflecting new blocking policies is not yet available.

The primary mission of the OIT Networking & Monitoring Services group is to ensure the correct and efficient operation of the campus network for the benefit of Princeton University faculty, staff, and students.

Sometimes our mission makes it necessary to deny network service for a customer's device. For example, we block devices that disrupt or degrade network service. We also block a device when its activity creates a security risk for others. At times we are directed to block service because a customer's network activity places the customer or the University at legal risk, or the activity is in serious violation of University policies.

This document provides an overview of the most common reasons we block devices, how customers are notified of these blocks, and the timeframe for removing such blocks once the customer has resolved the issue.

Contents


Why Are Devices Blocked by OIT Networking & Monitoring Services?

These are the most common reasons that lead to OIT Networking & Monitoring Services blocking service for a device:

Disrupting Service
When a device disrupts network service, we take appropriate action to restore normal service for the campus as expeditiously as possible. Usually this includes immediately blocking network service for the disruptive device.

Some common examples include devices that create a bridge loop, steal another's IP address, or act as rogue DHCP servers,

Degrading Service
Sometimes a device's activity isn't actually disrupting network service, but instead is only degrading network service. We take appropriate action to restore the quality of network service to an acceptable level, or to prevent the service quality from falling below an acceptable level.

Examples include devices that flood OIT DNS Service with excessive requests, keep asking OIT DHCP Service for new leases, or toggle their connection (link) OIT Ethernet Service up and down at excessive rates.

Depending on the degree of service degradation, it may be necessary to block the device immediately. Other times it may possible to give the customer some time to address the issue without blocking the device; if the problem continues, however, we will block the device eventually.

Impeding the Ability of OIT to Manage Network Service
On occasion a device is neither disrupting nor degrading network service, but is operating in such a way as to impede the ability of OIT to manage network service. Ultimately this can affect the quality of network service, because when OIT's ability to manage network service is impeded, we cannot respond expeditiously to detect and correct network problems.

Usually it possible for us to give the customer some time to try to fix the problem, or to disconnect the device until s/he can fix it. If the issue continues (the customer neither fixes the problem nor disconnects the device), we may block the device.

While uncommon, examples include devices that cause an excessive frequency of certain ordinary network events to be logged by network infrastructure. The logging of some ordinary is necessary to allow OIT to perform diagnosis and management of the network. A device that is responsible for an excessive frequency of these events can impede our ability to maintain usable logs.

Attacking or Compromising Other Devices
A device that appears to be the source of an attack or attempted attack on other devices may be blocked. Typically we block as soon as is practical.

Some examples include devices scanning the network to find targets for potential attacks, devices attempting to guess passwords, and devices attempting to flood targets with excessive traffic.

Persistent Attachment to Campus Network without Host Database Registration
Devices attached to the campus network must be registered in the Princeton University Host Database. (That document notes there are several exceptions, most notably devices properly using Temporary Visitor Wireless Network Access (TVWNA).)

When a device is persistently attached without being registered, OIT Networking & Monitoring Services will request that someone responsible for the device register the device or disconnect the device. If the device remains attached and unregistered, we block network service for the device after a suitable period.

Often these situations also come under the "degrading network service" category. This is because the device often is configured to obtain an IP address via DHCP or BootP. The device's ongoing DHCP or BootP attempts (which fail because the device is not registered) place unnecessary load on campus DHCP/BootP servers, and add unnecessary broadcast traffic to the network.

Violations
Various violations may lead to a device being blocked:

While these are the most common reasons we block network service for a device, it is not intended to represent a complete list. There may be less common situations that also are best addressed by blocking network service for a device. Each incident is handled individually, so we may take the most appropriate action as the circumstances warrant.


How Long Do We Wait Before Blocking a Device?

When a device is disrupting service, we usually must block the device immediately. The same is true when a device is attacking or compromising other devices. And we block immediately when there is a violation involved.

Often we do not need to block the device immediately. For example, if the device is only degrading network service (rather than disrupting it), and the degree of service degradation is not severe, it usually is possible to give the customer some time so s/he may fix the problem, or to disconnect the device until s/he can fix it.

For example, if the network can continue to perform acceptably while there are few devices degrading service in a particular way at any one time, we may be able to wait several days for the customer(s) to take action.

In these cases, the problem description OIT Networking & Monitoring Services provides often says something to the effect that "If the customer cannot correct the problem at this time, the customer should disconnect the device from the campus network until s/he can correct the problem."

If the issue continues (the customer neither fixes the problem nor disconnects the device), we block network service for the device. We do so because if we were to leave such matters unresolved, the number of such incidents would grow until their cumulative effect would be to degrade network service below acceptable levels.

How long we wait for the customer to address the problem before the device is blocked varies depending on the problem's nature. For many problems, we wait three business days from the time that a reasonable attempt was made to notify a responsible person of the problem.

For some "service is degraded" problems we permit the problem to continue only for a few hours before we must block the device. In such cases, we usually say so in the problem description. For example, we may say: "If the customer cannot correct the problem at this time, the customer should disconnect the device from the campus network until s/he can correct the problem. If the problem is still continuing at the end of the business day, network service for the device will be blocked." We recognize that it is unrealistic to expect most customers will be able to take action in so short a timeframe; we take this approach so that those customers who can take swift action may do so rather than be blocked.

How long we wait before blocking also may be influenced by whether the device has been the source of the same problem repeatedly in the past. For example, after a device has caused the same problem three times, further incidents may lead to the device being blocked promptly rather than giving the customer several business days to address the problem. Similarly, if a device caused the same problem in the past leading to it being blocked, was then unblocked because the customer informed OIT that the problem was resolved, and then the problem resumes quickly enough show that it had never been resolved, OIT will block the device promptly.

We also may block immediately when customer contact is impractical. That typically happens when the device is not registered in the Host Database (or the device's identity is unknown), and the specific physical location the device attached to the network also is unknown. If the device is not registered in the Host Database (or the identity of the device is unknown), we may have no customer contact information for the device. If the device is connected via OIT Ethernet Service but the OIT Ethernet switch port is missing from the OIT Network Atlas, then we have no specific location for the OIT Ethernet wallbox port. If the device is connected via a wireless service provided by OIT, then we have no specific location for the device.


How Do We Notify Customers?

When OIT Networking & Monitoring Services makes the decision to block the device, we ask the OIT Support and Operations Center (a.k.a. the OIT Help Desk) to notify someone responsible for the device.

We normally do so using Services Now (SN@P), the University's Information Technology Service Management system. SN@P assigns a unique "Incident ID" to the incident, and ensures that information about the incident is available round-the-clock to information technology support staff within OIT and in other University departments. This also makes it possible to refer the incident to information technology support staff in other University departments when appropriate.

How the OIT Support and Operations Center determines who best to notify can vary depending on the nature of the incident:

Once the OIT Support and Operations Center determines who should be notified, they determine how best to notify that person:

Exceptions to this notification approach include:


How Long Does it Take to Be Unblocked After the Customer Addresses the Issue?

We recognize that customers want to have their devices unblocked as quickly as possible. OIT Networking & Monitoring Services strives to handle "unblock" requests within one business day of the time we receive the request.

The problem description provided by OIT Networking & Monitoring Services usually states how the person handling the incident should advise Networking & Monitoring Services once the issue has been addressed, so that the block may be removed. For example, a SN@P incident we create might say:

Once the issue has been addressed, to have the block(s) reviewed for removal:
In this Services Now (SN@P) Incident, create an Incident Task for the
"Security & Monitoring" SN@P assignment group.  In that Incident Task, specify the action
you took to address the issue, and that you would like the block(s) reviewed for removal. 

So if the person responsible for the device notifies the OIT Support and Operations Center that the problem is fixed, the Support and Operations Center would follow those instructions to request that the block be reviewed for removal.

Alternatively, if the incident is being handled by departmental IT support staff who have access to SN@P, the departmental IT support staff would follow those instructions to request that the block be reviewed for removal.

This would result in an SN@P Incident Task (for the exising SN@P Incident) being created and routed to appropriate members of the OIT Networking & Monitoring Services group. Once OIT Networking & Monitoring Services has received that SN@P Incident Task, we strive to handle it within one business day. In support of this goal, we prioritize the handling of such "unblock" requests above many of the other tasks we perform.

Be aware that:

The "one business day turnaround" is specific to unblock requests for those blocks initiated by OIT Networking & Monitoring Services. It doesn't apply to other kinds of requests submitted to OIT Networking & Monitoring Services. We prioritize "unblock" requests above many other kinds of requests.


Appendix: How to Request Removal of a Block that Was Installed as Part of an OPM Ticket

On July 11 2016 OIT replaced "OPM" (its previous problem tracking system) with SN@P (Services Now at Princeton). OPM tickets which existed at that time were not automatically copied into SN@P. On October 31 2017, OIT retired read-only access to OPM.

Many devices which had been blocked for any of the reasons above were still blocked at the time OPM was replaced with SN@P. Typically, each block appeared in an OPM ticket with language such as Once the issue has been addressed, please transfer this OPM ticket to the OPM NET-INCOMING queue to have the block(s) reviewed for removal. It became impossible to follow those instructions once OPM was replaced with SN@P.

To continue handling such an incident which appeared in an OPM ticket (for example, to request removal of the block once the issue is addressed), someone needs to copy the contents of the OPM ticket into SN@P, then continue handling the issue in SN@P.

Normally the person who copies the incident from OPM into SN@P should be someone who would have handled it previously in OPM, or would handle the incident had it originated in SN@P. That's typically a person who provides support involving the device. Often this is departmental SCAD/DCS staff. When departmental SCAD/DCS staff aren't the people providing the support for the device or conveying the unblock request, it might be OIT Hardware Support staff (when they are conveying the customer's unblock request), OIT Client Software Support staff (if they are conveying the customer's unblock request), or OIT Support and Operations Center staff (when none of the above are providing the support, and the customer contacts the OIT SOC). Whoever performs this operation needs to be a SN@P "Fulfiller."

The person copying the contents of the OPM ticket into SN@P should follow very specific instructions in this situation. Instructions are in Knowledgebase article KB0010044: Procedure for resolution of devices degrading/disrupting Network Service originally reported in OPM.

(Having trouble accessing that article? Because that's a "Technical Article" (not a "General Article"), access is limited to SN@P fulfillers. If you have not already logged into SN@P as a fulfiller, you will be unable to view the article; SN@P may instead respond Knowledge record not found. In that case, login to SN@P, then view Knowledgebase article KB0010044.)

Following those instructions will result in creating a SN@P Incident containing a copy of the original OPM ticket's contents followed by the unblock request. That SN@P Incident will be automatically routed by SN@P to the OIT Support and Operations Center. The OIT SOC will then review the unblock request, and if appropriate, will update the SN@P Incident to create an Incident Task to pass the unblock request to the "Security & Monitoring" SN@P assignment group.

From that point, handling of the unblock request proceeds as described above in How Long Does it Take to Be Unblocked After the Customer Addresses the Issue?

The turnaround time for these requests is significantly longer than one business day. This is because OIT Networking & Monitoring Services staff will need to re-assemble the information from the original OPM ticket in which the block was installed, and the later SN@P ticket containing the request to remove the block.

Please do not use other mechanisms (phone calls, email, unrelated SN@P Incidents/Requests/Inquiries) to request the removal of the block. They will result in much longer processing time. You may then simply be asked to follow the procedure above, or someone else will have to perform the procedure above on your behalf.

Copy the Contents of an OPM Ticket into SN@P Only Once

Such an OPM ticket's contents should be copied into SN@P just once.

Once it's already copied into SN@P, any further activity should be using the SN@P Incident that was created. If someone were to copy the contents of the OPM ticket into SN@P a second time, the second SN@P incident would be missing all activity which has happened in the first SN@P Incident since the time the OPM ticket's contents was copied into the first SN@P Incident.

As a result, before copying an OPM ticket's contents into SN@P, you will want to check SN@P to be sure the OPM ticket wasn't already copied into SN@P. If it was, continue using the existing SN@P Incident; it's more recent than the old OPM ticket.


Appendix: How to Request Removal of a Block that Was Installed as Part of a SN@P Ticket That Has Since Been Closed

When OIT Networking & Monitoring Services installs a block as part of a SN@P ticket, the SN@P ticket likely contains instructions such as:

Once the issue has been addressed, to have the block(s) reviewed for removal: In *this* Services Now (SN@P) Incident, create an Incident Task for the "Security & Monitoring" SN@P assignment group. In that Incident Task, specify the action you took to address the issue, and that you would like the block(s) reviewed for removal.

If someone closes the SN@P Incident ticket while the block remains present,the block is not reviewed for removal.

To continue handling that SN@P ticket to have the block removed, a SN@P fulfiller will need to take action.

The SN@P fulfiller in this case is typically a person who provides support involving the device. Often this is departmental SCAD/DCS staff. When departmental SCAD/DCS staff aren't the people providing the support for the device or conveying the unblock request, it might be OIT Hardware Support staff (when they are conveying the customer's unblock request), OIT Client Software Support staff (if they are conveying the customer's unblock request), or OIT Support and Operations Center staff (when none of the above are providing the support, and the customer contacts the OIT SOC).

If the existing SN@P Incident ticket (where the block was installed) can be re-opened, re-open that SN@P Incident ticket, then follow the instructions that were in that ticket to request that the blocks be reviewed for removal.

Otherwise, if that existing SN@P Incident ticket can no longer be re-opened (SN@P does not permit re-opening Incident tickets after they have been closed for a some number of days), a SN@P fullfiller will need to open a new SN@P ticket to continue the handling of the original issue. That SN@P fulfiller should follow the instructions in Knowledgebase article KB0010540: How to resolve a "Device Degrading Network Service" incident after SN@P ticket is closed.

(Having trouble accessing that article? Because that's a "Technical Article" (not a "General Article"), access is limited to SN@P fulfillers. If you have not already logged into SN@P as a fulfiller, you will be unable to view the article; SN@P may instead respond Knowledge record not found. In that case, login to SN@P, then view Knowledgebase article KB0010540.)

Following those instructions will result in creating a new SN@P Incident. That SN@P Incident will be automatically routed by SN@P to the OIT Support and Operations Center. The OIT SOC will then review the unblock request, and if appropriate, update the SN@P Incident to create an Incident Task to pass the unblock request to the "Security & Monitoring" SN@P assignment group.

From that point, handling of the unblock request proceeds as described above in How Long Does it Take to Be Unblocked After the Customer Addresses the Issue?

The turnaround time for these requests is significantly longer than one business day. This is because OIT Networking & Monitoring Services staff will need to re-assemble the information from the original SN@P ticket in which the block was installed, and the later SN@P ticket containing the request to remove the block.

Please do not use other mechanisms (phone calls, email, additional unrelated SN@P Incidents/Requests/Inquiries) to request the removal of the block. They will result in much longer processing time. You may then simply be asked to follow the procedure above, or someone else will have to perform the procedure above on your behalf.


Appendix: Why Don't Networking & Monitoring Services Staff Contact Customers Directly When We Block a Device?

Customers sometimes inquire why OIT Networking & Monitoring Services asks the OIT Support and Operations Center to notify customers when their devices are causing a problem, rather than having OIT Networking & Monitoring Services contact customers directly:

We recognize that there is some validity to the observations above, however, there are reasons that OIT Networking & Monitoring Services asks the OIT Support and Operations Center to notify customers when their devices are causing a problem. These reasons include:

For those reasons, OIT Networking & Monitoring Services relies on the OIT Support and Operations Center to notify customers when we discover devices that are causing problems.


Appendix: Why Don't Networking & Monitoring Services Staff Contact Customers Directly When We Unblock a Device?

Customers sometimes inquire why OIT Networking & Monitoring Services asks the OIT Support and Operations Center to notify customers when we remove blocks, rather than having OIT Networking & Monitoring Services contact customers directly:

We recognize that there is some validity to the observations above, however, there are reasons that OIT Networking & Monitoring Services asks the OIT Support and Operations Center to notify customers after blocks have been removed. These reasons include:

For those reasons, OIT Networking & Monitoring Services does not remove blocks until being notified that the issue has been addressed, and the group relies on the OIT Support and Operations Center to notify customers after blocks are removed.


A service of OIT Networking & Monitoring Services
The Office of Information Technology,
Princeton University
Last Updated: December 27 2018