More Sysadmin Tips

Tuesday, 2007-08-21; 19:26:00



Back in March, I offered 5 tips for sysadmins that have to manage and play tech support for a lab or a whole site of Macs. Today I'm back to offer five more.

The main point of these entries is actually for my own benefit. Posting them here means that I can access these tips at anytime from anywhere that has interweb access and a web browser. It also means that I don't have to reinvent the wheel every time I come back to these problems. Inevitably these scripts will be lost in a computer shuffle or accidental deletion; posting them here makes it much less likely that they'll be lost into random oblivion. If all else fails, I can even consult the Internet Archive Wayback Machine if somehow I delete them from my website.

But, of course, what benefits me also benefits others on the internets. Google will index these tips, and then hopefully they'll provide some relief for others who are banging their heads against the wall trying to fix a certain problem. I know Google has helped me out many a time, so I'm happy to do the same. Hopefully I can also help separate the wheat from the chaff in terms of the resources out there that provide more detailed information.

In fact, my "5 Tips for Mac OS X Sysadmins" entry is the fourth most viewed entry ever on Technological Supernova, and is the third most viewed entry if you're only counting the time since I actually posted the entry. It is actually viewed almost half as many times as the main page of Technological Supernova itself: the entry page has been viewed 991 times since it was posted, compared with 2029 views of the main page. So hopefully at least a couple of those 991 pageviews were useful. :)


1. NetInstall Troubleshooting

NetInstalling is a godsend... when it works. It allows you to just hookup a bunch of Macs to your local network, boot them from the server, and essentially copy over a disk image that's hosted on the server. This not only means that you can install Mac OS X without having to pop a DVD in every Mac, but you can pre-install all the necessary apps for your Macs on the disk image which will copy over to the other Macs through NetInstalling. Yes, you can do this if you have a FireWire cord, but again, if you have a lab of 30 Macs, you'll need quite a few FireWire cords, and you still won't be able to do them all at once.

Unfortunately, there are always a few hiccups that come with things like this, and NetInstalling is no different. I don't hope to provide any sort of comprehensive NetInstall troubleshooting guide; that's been taken care of by Mike Bombich. The thing is, his write-up is incredibly detailed, and it may actually cause you to bang your head even more than is necessary.

Here are the problems I encountered.

First, don't put Mac OS X 10.4.9 or 10.4.10 on your disk image. Apparently there's a bug in the latest versions of the System Image Utility that cause these images to fail when NetInstalling. Be aware: it's OK to upgrade your server to Mac OS X 10.4.10, but not the client that you clone to create your hosted disk image.

This is going to be a problem, though, if you have really recent Macs that require 10.4.9 or 10.4.10 to boot up. In these cases, log in to the server, and find the NetBoot sharepoint. This is usually located at /Library/NetBoot/NetBootSP0/ . Open the desired .nbi file, and double click on the Install.dmg file. Then, copy the framework located at /System/Library/Frameworks/CoreVideo.framework from a 10.4.9 or 10.4.10 system to the same location on the disk image. Then eject the disk image. (Be aware, if your image is universal, you'll need the universal framework from an Intel machine. If your image is for PPC Macs only, feel free to copy the framework from a PPC machine.)

Second, don't use Disk Utility to create a disk image of a machine before creating the NetInstall image with System Image Utility. This'll make the process take twice as long: System Image Utility recreates an entirely new disk image from the one you already made. If you use System Image Utility to directly clone a client machine, you're only creating one disk image, not two, so the time it takes is significantly reduced.

(Also, note that you can edit images after you've created them. Open System Image Utility, and click on the "Images" button in the toolbar. I completely overlooked that for a while.)

Third, you don't need the DHCP service enabled on your server for NetInstalling to work. Some earlier documentation and pages on the interwebs indicate that you do need DHCP enabled on the server, but that's not true with Mac OS X 10.4 Tiger Server. If there's already another router or server doling out IP addresses via DHCP, you can leave it off and still NetInstall.

Fourth, if your PPC Macs don't get past the blinking globe stage of NetInstall, try making an image and NetInstalling an Intel Mac. If the Intel Macs NetInstall fine, you're probably encountering a weird interaction between NetBoot and Cisco switches. (Info derived from Bombich's troubleshooting page.)

The problem here is that a firmware setting in older PPC Macs is conflicting with newer Cisco switches. To rectify this, send the following command out to all your PPC Macs via Remote Desktop: sudo nvram default-bootp-vexts="%00" . Restart the computers and then hold down the 'n' key to NetBoot, and you should now get past the blinking stage.

If the PPC Macs in question don't have Mac OS X installed or otherwise can't get to a point where they can be administered via Remote Desktop, fear not; you'll just have to implement this fix manually. Restart them, and hold down Command-Option-O-F to drop into the Open Firmware prompt. Type setenv default-bootp-vexts %00 and then press return. The console should print a little "ok" after your command and give you another prompt. Now type mac-boot , press return, and immediately hold down the 'n' key to start the NetBoot process.

Also be aware: this setting will get erased next time you zap the PRAM by holding down Command-Option-P-R on startup.

Fifth, if NetInstall stalls at a blue screen with a spinny progress indicator or the spinning rainbow cursor without even getting to the Network Install screen, something on the local hard drive could be interfering with the installation process. Force shut down the computer, connect it to another Mac via target disk mode, and erase the local hard drive. Then restart and try to NetInstall again. I've found that this often occurs if a NetInstall fails during the actual installation process, whether due to an error or due to a power failure. Erasing the local hard drive immediately fixes the issue.


2. Scripting Open Directory Binding Revisited

Last time, I published a script that would automate the process of binding a Mac to an LDAP server. A few modifications are in order.

First, some system update or weird confluence of settings has caused the logging in portion of the script to stop working for me. (Discussion at Mac OS X Hints finds that other people are having similar problems.) I'm not sure why or how, but the "keystroke return" line after putting in the login screen password now causes the password to simply become highlighted. The only viable solution I've found is to duplicate that "keystroke return" line a couple times. Even then, it's finicky, because it's not consistent on which "keystroke return" line finally causes the login. The upshot is that you can sometimes have an extra "keystroke return" line in there, and it fires right as the Mac is switching users, causing that last line to fail and stop the script.

That means you'll have to break up this script into two portions: the login portion and the Directory Access portion. Here's the login portion:

sudo touch /private/var/db/.AccessibilityAPIEnabled
osascript <<EndOfMyScript 
tell application "System Events"
        keystroke "login" 
        delay 2
        keystroke tab
        delay 2
        keystroke "password" 
        delay 2
        keystroke return
        keystroke return
        keystroke return
        keystroke return
        keystroke return
end tell
EndOfMyScript


This script will probably fail if you push it out via Apple Remote Desktop, but it doesn't matter since the login will complete successfully. Also note: be sure to replace "login" with the full user name, not the short user name. This ensures that the login goes properly whether or not the login window is set up to show a list of users or just username and password fields.

The second part of the script also needed to be slightly changed: a Mac OS X update changed the position of the "New…" button in Directory Access, making the previous version fail. Also, I added a bunch of "delay" lines in order to accomodate for slower computers. Here's the second part of the script:

osascript <<EndOfMyScript 
tell application "System Preferences" to activate
delay 5
tell application "System Events"
        tell process "System Preferences"
                click button "Sharing" of scroll area 1 of window "System Preferences"
                delay 5
                set computer_name to (get value of text field 1 of window "Sharing")
        end tell
end tell
tell application "Directory Access" to activate
delay 5
tell application "System Events"
        tell process "Directory Access"
                select row 5 of table 1 of scroll area 1 of tab group 1 of window "Directory Access"
                set connection_method to (get value of text field 1 of row 5 of table 1 of scroll area of tab group 1 of window "Directory Access")
                if (connection_method as string) is equal to "LDAPv3" then
                        click button 1 of tab group 1 of window "Directory Access"
                        delay 5
                        click button 1 of sheet 1 of window "Directory Access" -- change "button 1" to "button 3" for earlier versions of Mac OS X
                        delay 5
                        click text field 1 of group 1 of window "New LDAP Connection"
                        delay 5
                else
                        display dialog "An error occurred: selected row is not LDAP." buttons {"OK"}
                end if
        end tell
end tell
tell application "Directory Access" to activate
tell application "System Events"
        tell process "Directory Access"
                keystroke "address-to-directory-server.domain"
                delay 2
                click button "Continue" of window "New LDAP Connection"
                delay 5
                set value of text field 3 of group 2 of window "New LDAP Connection" to computer_name
                delay 2
                set value of text field 2 of group 2 of window "New LDAP Connection" to "dirlogin"
                delay 2
                keystroke tab
                keystroke tab
                keystroke "dirpassword"
                delay 5
                click button "Continue" of window "New LDAP Connection"
                delay 5
                click button "OK" of window "New LDAP Connection"
                delay 2
                click button "OK" of sheet 1 of window "Directory Access"
        end tell
end tell
tell application "Directory Access" to quit
tell application "System Events" to log out
EndOfMyScript
sudo rm /private/var/db/.AccessibilityAPIEnabled


I did get a comment on my previous entry that supposedly allows you to bind computers to a server via the "dsconfigldap" command. Trouble is, this doesn't actually bind the client to the server as it should and as it states in the man pages, at least not in the Mac OS X sense. After doing this command, you don't get the "Other..." option in the list of users at the login window. So for now I'm sticking with the GUI scripting method.


3. Setting common network and system settings

There are many system settings that are difficult to change en-masse, and many of these apply to network settings. A couple examples would be: DNS servers, sleep times, Bonjour name, waking on network access, etc. You could try to find the specific preference file to propagate, but you'd probably overwrite several other settings in the process.

Fortunately, Apple Remote Desktop provides two cool little utilities that facilitate access and modification of many system and network settings. They're called "systemsetup" and "networksetup", and they live inside the ARDAgent application that allows Remote Desktop to control any Mac. Best of all, this utility is included with any installation of Mac OS X, even if Remote Desktop wasn't specifically installed.

In a Terminal window, type or paste in sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup to see a list of possible commands and their explanations. Similarly, sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/systemsetup -help will show you all the system settings you can change.

My particular favorites:

  • The following command detects any new network hardware and creates new network services for them, and then lists all available network services. These are the entries that appear in the Network Port Configurations list in the Network system preference pane. Why it's useful: the list of network interfaces is stored in a file at /Library/Preferences/SystemConfiguration/NetworkInterfaces.plist . Unfortunately, this stores the MAC addresses of these interfaces as well, which change from machine to machine. Propagating this file to a bunch of Macs won't work, then, so this command helps to setup network interfaces en masse as you see fit.

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -detectnewhardware; sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -listallnetworkservices

  • This one works well with the previous one: it allows you to change the order of network services. Run the previous command to detect all valid pieces of hardware and list them for you. Then change this command such that you list all of the services provided by the previous command, and in the order that you desire. Allows you to do this en masse.

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -ordernetworkservices "Built-in Ethernet" "AirPort" "Built-in FireWire" "Bluetooth"

  • This one changes the Bonjour name to match the computer name as listed in the Sharing preference pane. Why it's useful: when using Apple Remote Desktop to rename computers, it doesn't change the Bonjour name! Argh! This allows you to resync up the Bonjour name and computer name. Watch out: this will fail if your computer name has underscores or spaces.

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/systemsetup -setlocalsubnetname "`sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -getcomputername`"

  • What it does: changes DNS servers of a specific network interface. Good in combination with the first command of this set. Why it's useful: you can change DNS servers en masse; string a couple of these commands together, and you can change the DNS servers for all network interfaces of many machines en masse as well. Yay for automation!

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup -setdnsservers "Built-in Ethernet" 10.0.1.1 10.0.1.2 10.0.1.3;

  • What it does: makes the machine wake if it detects network activity. Why it's useful: allows you to wake up machines via Remote Desktop on your local subnet.

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/systemsetup -setwakeonnetworkaccess on

  • What it does: changes the sleep delay. Why it's useful: allows you to change the sleep delay en masse (like, duh!). Use "-setcomputersleep" for CPU sleep delay, "-setdisplaysleep" for display sleep delay, "-setharddisksleep" for hard disk sleep delay, or just "-sleep" to set all three at once. String a few of these commands together and you can set the various delay times to different settings all at once.

    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/systemsetup -setcomputersleep 30;

Everlasting gratitude to John C. Welch for his entry regarding networksetup which enlightened me to these two tools. Yes John, your entry did get picked up by Google and it did help me out. :)


4. Propagating AirPort settings

This one was my pet project for today. I needed to let 90 laptops know of a particular closed network, get it to automatically connect to this network on login, and not allow the changing of wireless networks without an administrator password. Most of this stuff is set in the "AirPort" tab of the AirPort network interface in the Network system preference pane.

Also, most of the AirPort settings are stored in a single preference file without any other network settings. This file, located at /Library/Preferences/SystemConfiguration/com.apple.airport.preferences.plist can safely be transferred to other computers and dropped in the same location.

Unfortunately, this doesn't take care of a few things. First, the password for the closed network isn't stored in this file. The setting in the "By default, join" popup menu is also not stored here. Also, the setting that requires an admin password to "Create Computer-to-Computer networks" isn't stored in that file, either, although curiously it seems that requiring one to change wireless networks is.

Sadly, systemsetup and networksetup don't facilitate the modification of these prefs, either, so back to GUI scripting it is! Here's my script for taking care of these last few settings, after first propagating the com.apple.airport.preferences.plist file:

sudo touch /private/var/db/.AccessibilityAPIEnabled
osascript <<EndOfMyScript 
activate application "System Preferences"
delay 5
tell application "System Events"
        tell process "System Preferences"
                click button "Network" of scroll area 1 of window "System Preferences"
                delay 5
                click pop up button 2 of window "Network"
                click menu item "AirPort" of menu 1 of pop up button 2 of window "Network"
                delay 2
                click pop up button 1 of group 1 of tab group 1 of group 1 of window "Network"
                click menu item "Preferred networks" of menu 1 of pop up button 1 of group 1 of tab group 1 of group 1 of window "Network"
                delay 2
                select row 1 of table 1 of scroll area 1 of group 1 of tab group 1 of group 1 of window "Network"
                click button "Edit"  of group 1 of tab group 1 of group 1 of window "Network" 

                delay 2
                keystroke tab
                delay 1
                keystroke "closedNetworkPassword"
                delay 1
                click button "OK"  of sheet 1 of window "Network" 
                delay 2

                click button 3 of group 1 of tab group 1 of group 1 of window "Network"
                delay 2
                click checkbox "Create Computer-to-Computer networks" of sheet 1 of window "Network"
                delay 1
                click button "OK" of sheet 1 of window "Network"
                delay 3
                click button "Apply Now" of window "Network"
        end tell
        log out
end tell
EndOfMyScript
sudo rm /private/var/db/.AccessibilityAPIEnabled


This script assumes that you only have one preferred closed network, since it simply selects the first in the list of closed networks. Modify at your leisure.


5. Resetting the system keychain

I noted in the previous tip that the password for the closed network isn't stored in the AirPort preference file. That's because it's stored in the system keychain: this allows any user to access the network from that machine without needing to re-enter the password.

I first tried to simply propagate the system keychain file to 30 of laptops, located at /Library/Keychains/System.keychain . This seemed to work for the first 30, but it produced disastrous results for the second 30.

Once I propagated the system keychain to the second set of laptops, they started asking me for the password to the system keychain when setting the password of the closed network. Thing is, the system keychain doesn't have a password. A blank password doesn't work. There's nothing you can put into this password prompt that pops up that'll allow you to open the system keychain. In Keychain Access, you need to put an administrator login and password to access the system keychain, but in the panel I was having trouble with, it only asks for a password.

Problem is, once you get into this situation, it's hard to get out. Deleting the system keychain file doesn't solve the problem. Copying another system keychain file from another computer doesn't solve it. Keychain First Aid accessed from the Keychain Access application doesn't check the system keychain file. In short, you're screwed at this point if you don't know the magic incantation.

Here's the command to salvation, taken from an Apple discussions page: sudo /usr/sbin/systemkeychain -C -v . You'll probably want to precede it with this command: sudo rm /Library/Keychains/System.keychain which deletes the problematic system keychain.

Moral: never propagate the system keychain file to other machines.


Technological Supernova   Tips   Older   Newer   Post a Comment