Friday, April 25, 2014

Patching Computers running Deep Freeze using SCCM in Higher Ed

Deep Freeze introduces some issues that need to be accounted for when planning a computer lab maintenance schedule with Configmgr. At the university I work for we have implemented the following solution.

Our lab maintenance time is weekly on Friday mornings from 2:15AM to 6A.M. This time was chosen because its when we could have all the computers labs out of service with the least impact on the students. Some disciplines, notably the science and engineering students have 24 hour access to their computer labs which can create difficulties in establishing a standard maintenance time for all computers across the empire. We would like to have had multiple maintenance times a week, but it wasn't possible across the whole campus and for simplicity's sake, currently we have not broken up labs into different maintenance windows.





2:15 AM - Computers turn on! We hope. We try to ensure that the computers are on during the maintenance window in the following two ways.

First, we have pushed packages that modify the BIOS on those computers that support it to turn on at the appropriate time. We have found that WOL is not always reliable in our environment, so configuring the BIOS on computers that support it is desirable. This has enabled us to wake a large number of our PC's for the maintenance time. We are working on getting to a 100% success rate, either by updating the BIOS to versions that support auto turn on, or replacing the computers with ones who's BIOS support auto turn on. We use Dell CCTK to configure the BIOS on Dell's during deployments, Dell OMCI to modify the BIOS on already deployed Dell systems, and the HP Bios Configuration Utility for HP's. These utility's are really cool, they make it so you do not have to visit the computers to configure the BIOS's.

Second, we send a Wake On Lan command from the Configmgr server to the computers in the maintenance window collection. The way we do this is by deploying a package at 2:15AM on Friday mornings. We set this deployment to ignore maintenance windows. Since at this time the computers are still "frozen" by Deep Freeze, we do not want the package to do anything, we are just gaming Configmgr to send a WOL signal by deploying the "do nothing" package. An example of this deployment is here.

2:25AM - Once the computers are on, Deep Freeze configuration is set to reboot the computers in "thawed" mode.

2:30AM - Configmgr then puts the computers into the maintenance window at 2:30AM. All the computers we are updating are a member of a collection with a maintenance window set. On the properties of the collection, under the Maintenance Window tab you can schedule this. Below is ours which lasts from 2:30AM to 5:30AM every Friday Morning.



2:35AM - a task sequence is scheduled to run every week. This task sequence updates the computers and ensures that they are getting any reboots needed before Deep Freeze "freezes" the computers at the end of the maintenance time. In order to ensure that the Task Sequence is updating the computers during its "Install Software Updates" steps and is not being interfered with by updates coming from Configmgr outside of the task sequence, our software updates are deployed as "Available" to the collection of lab computers we are updating, and not "Required". Updates will only be installed to a collection if they are deployed to that collection, either available or required. If the collection does not have any updates deployed to it, then the "Install Software Updates" step of the Task Sequence would not install any updates. If we made the updates "Required" some of them might try to install during the 5 minute window between the start of the Maintenance Window at 2:30AM and the start of the Task Sequence at 2:35AM.

The Task Sequence looks like this, I will go into more details of each step in another post.


5:30AM - the Configmgr Maintenance Window ends, we may push this back to 5:45 to give another 15 minutes of maintenance time.
6:00AM - Deep Freeze shutdowns the computer, and the next time it boots it will be frozen.

Pushing update packages vs. the control of a task sequence

We use to just push update packages at the lab collection to update the computers instead of updating via a task sequence. We found that some updates would need a reboot to finish installing but not force a reboot, and this could cause problems when using Deep Freeze. When Deep Freeze "froze" the machine at the end of the maintenance time and the computer would boot up in the morning, the update would continue updating, and then reboot while updating. Since the machine was frozen, this would put it into a continuous loop of rebooting and updating. Not an ideal situation for a classroom or lab. I had originally attempted to fix this by running a reoccurring script that would force a reboot after the Configmgr maintenance time, but before Deep Freeze "froze" the computer. This ended up not working and the errors in the logs seemed to suggest the computer was waiting for a reboot before it would run any more deployments, including the deployment of the reboot script. The task sequence appears to handle reboots much better and gives greater control over the order that software and updates get installed and when commands are run, like the hardware inventory command that runs at the end of the task sequence.

6 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. Hello David,

    Thanks for the details of your solution. I have been trying to implement your solution and had some limited success but have run into one issue.I have set a very long maintenance window which I thought should be more than enough to handle this. At 11:30PM Deep Freeze puts my labs into a thawed Batch File State (the batch file just runs a command to run a Machine Policy Retrieval) and the remain thawed until 7:00AM. The machines are in a collection that has a maintenance window of 12:00AM - 6:30AM which should give the machines time to get their policy renewal and realize that the have a task sequence to run and end with enough time for Deep Freeze to gracefully freeze. The task sequence itself has run time limit of 5 hours (I set the run time to less than the maintenance window to give the machines extra time to start the task sequence without going outside the service window.

    The problem that I have been trying to fix is that the majority of my lab machines don't start the task sequence in time and are left in a "Waiting for a Service Window" state. Originally I was getting about a 5% success rate, since I switched to the Batch File Thaw period, I have gotten it up to a 40% success rate. Did you run into this issue and, if so, can you tell me how you managed to fix it?

    Thanks,
    Ryan

    ReplyDelete
    Replies
    1. reposting so its a reply and not a new post :) i always see to do that.
      Hello Ryan,

      There are two things I would try. I would change your Maximum allowed run time to a shorter time, 5 hours means it will not attempt to run or rerun the task sequence after 1:30AM. You do not have to worry about going over the maintenance window as each deployment or update in the task sequence has its own maximum allowed run time. I believe its 15 minutes for each individual update. As an example I have my task sequence's maximum run time at 60 minutes.
      The other thing to consider is when did you create your deployment schedule? If you created it before the March daylight savings time change, its possible it is trying to run an hour later than you are planning. I recreate any of my reoccurring scheduled tasks twice a year after the time changes.
      http://myitforum.com/myitforumwp/2012/11/14/mischief-automatic-deployment-rule-evaluation-and-daylight-savings-time/

      I would try those two things, see how it goes then start investigating further if the rates dont improve.

      I now use a scheduled task in the Deep Freeze console to send a reboot to the computers at 5:45am, after the configmgr maintenance window ends, and before DF ends the thaw period. This helps insure that the updates that MS occasionally release that require double reboots do not get the computers stuck in a reboot loop.
      -Dave

      Delete
  3. The whole campus and for simplicity's sake, currently we have not broken up labs into different maintenance windows.pc building guide

    ReplyDelete
  4. The deep you dig into the subject and give us the accurate data is appreciable.
    best desks

    ReplyDelete