August – 2012 – www.gavinwill.me.uk

Automatic Network Configuration Version Control with Rancid and Procurve Switches

Rancid is a really handy bit of software that can automatically check for configuration changes of routers and switches, email notification of these changes whilst maintaining the history in SVN or CVS.

When I set this up it just worked for our Cisco switches however I intially had a problem with the Procurve Switches. After adding the switches to the router.db with the following syntax

switchhostname:hp:ip

And then manually kicking off rancid with the command

./bin/rancid-run

I noticed it took the command ages to run and not much happened. Examing the logs in /var/log/rancid/GroupName.Date I saw lots of errors with timeouts. Investigating further this was an issue with hpuifilter

Error: ssh failed: couldn't execute "hpuifilter": no such file or directory

hpuifuilter is in the /rancid/bin/ folder however the path was not specified. With running Rancid on an Ubuntu 12.04 server I simply modified the global path in /etc/environment/ and after this it appeared to work a little better but still not pull in the config from the procurve switches.

I then tried a manual run of hlogin to see if it could simply pull the time from the switch with the command

./hlogin -c 'show time' switchhostname

However this would not get past the Procurve welcome screen

A resolution to this is to specify autoenable in the .cloginrc file.

add password switchhostname {password} {enpassword}
add autoenable switchhostname 1

After that simply set a cron job to run rancid-run at your preffered interval, configure postfix and your email groups and thats it. You will now have all the configuration history and automatically be emailed when there is any configuration change.

Automatic Notification for Active Directory Account Lockouts

The Task Scheduler in Windows 2008 is vastly improved from previous versions of Windows Server. One feature that I really like is that you can trigger tasks from events showing up in the logs. Since account lockouts are listed as Event-ID 4740 we can create a task that emails the IT department or helpdesk as soon as that ID enters the security log. The IT department therefore are aware there is an issue and can pre-empt the user asking for help. It can also assist in being notified when there is a brute force attack being made.

First we need a Powershell script that can email the information from the Security log about the lockout to the IT department.

$Event=Get-EventLog -LogName Security -InstanceID 4740 -Newest 1
$MailBody= $Event.message
 
$MailSubject= "User Account locked out"
$SmtpClient = New-Object system.net.mail.smtpClient
$SmtpClient.host = "smtp.domain.com"
$MailMessage = New-Object system.net.mail.mailmessage
$MailMessage.from = "AccountLockout@domain.com"
$MailMessage.To.add("alerts@domains.com")
$MailMessage.IsBodyHtml = 1
$MailMessage.Subject = $MailSubject
$MailMessage.Body = $MailBody
$SmtpClient.Send($MailMessage)

Save this powershell script in a script directory that is accessible from your server and then simply create a task within the Task Scheduler. The Trigger is On an event

Using the Secuirty log and Event ID 4740. The Action is then to run the powershell script to find the information in the log and to email it to the IT department. For starting powershell scripts you need to tell the task scheduler to Start a Program and to start Powershell with the argument being the location for where the script is saved.

Now simply create a test user in Active Directory and try and login to a computer with incorrect password. When the amount of failed logins are reached depending on your account lockout policy you will then shortly see the email alert come in to notify that an account has been locked out.

Vmware Enhanced VMotion Compatibility Advantages and Disadvantages

I was asked about VMware EVC the other day and the advantages and disadvantages it may have. I had not used Vmware clustering with only using ESXi free and so was stumped at this question. Naturally I had to find out the answer. EVC alleviates the issue of vMotion compatibility between hosts that have different CPU generations in a cluster. EVC automatically configures server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. As with almost all things Vmware you need to ensure your hardware is on the Hardware Compatibility list to ensure you are running supported EVC CPU Types. You can search the Hardware Compatibility List to verify if your processor models are listed.

There is also the restriction the EVC does not allow for migration with vMotion between Intel and AMD Processors. Other points you need to be aware of is the BIOS settings of these hosts need to Enable Hardware Virtualization and Execute Protection.

It appears that the enabling EVC in your organisation will have benefits in the long run however planning is critical to ensure ensure EVC will work with your current hardware, a planned maintenance is avialible to configure EVC since all virtual machines need powered off (or start with an empty cluster), the Bios settings of the hosts are correct and that if you do have a mix of Intel and AMD you are aware that you cannot vMotion between the 2 different types.

Simple Nagios Check for Windows Share Mount in Linux

I had a requirement to ensure that a linux server was mounted to a windows share that was used for backups. Adding this check into Nagios was incredibly easy with a custom script and use the check_by_ssh command.

First I needed to create a script that would check if the mountpoint existed and then report back an exit code of 2 if it is mounted and an exit code of 0 if it was not mounted so Nagios can report accordingly.

 
test "`mount | grep windows.share`" ==  ""
if [ $? == 0 ]; then
echo " Windows Share Not Mounted.  Backups will fail "
exit 2
 
else
echo " Windows Share Mounted "
exit 0
fi

This script was then saved in /usr/local/bin as check_mount.sh and made executable.

On the Nagios Server I then created the simple custom command to run this script.

##############################################################################
# CUSTOM Check Linux Mounted to Backup Drive
#
##############################################################################
 
define command{
command_name    check_mount
command_line $USER1$/check_by_ssh -H $HOSTADDRESS$ -t 45 -C /usr/local/bin/check_mount.sh
}

Finally you then need to define the service for the check to occur.

define service{
        use generic-service
        host_name remoteserver
        service_description Backup Drive Mounted
        check_command check_mount
}

With the command now defined it is really easy to add any further monitoring of other linux servers that have a windows share mounted by simply copying the script over, amending it to suit and adding the extra host to the service description.

Regular Expressions in Nagios

I am a massive fan of Nagios. Whilst it has an initial learning curve it can be an incredibly powerful system. Assuming you have a standardised naming structure for your servers, switches and other infrastructure you can use regular expressions to create a hostgroup that will cover all your devices.

define hostgroup {
    hostgroup_name  Domain Controllers
    alias           All Domain Controllers
    members ^.*dc[0-5]$,^.*(dev|prd|)$
}

If you run a logical structure and have lots of switches or servers this can be a really easy way to ensure you are monitoring all servers or switches and can easily be expanded by simply changing the regex.

Analysing Memory Dump Files with WinDbg

WinDBG which is a part of the Microsoft Windows SDK is a great tool for analysing Blue Screen Of Death memory dumps to find out what was the cause of a crash.

Typically if there is a BSOD the memory dump will be saved in C:\Windows\Minidump. Copy this file from the faulty computer (you may need to go in via safe mode to obtain the file if it crashes so often you cant even login) onto a computer that you will be installing / running WinDbg on.

On a working computer install WinDBG from the Microsoft Windows SDK

When installed start WinDbg but before we open up the memory dump we need to first specify the symbol search path. To do this simply goto File > Symbol File Path and specify the following

 SRV*c:\temp*http://msdl.microsoft.com/download/symbols

Now simply goto File > Open Crash Dump and browse to where the memory.dmp is saved. To obtain more verbose infomation on the problem run the command

 !analyze -v

In this instance I had a laptop that would blue screen every 20 minutes. This was due to a Driver Power State Failure on usbccpg.sys. The resolution in my case was simply to boot into safe mode and remove the drivers for the USB subsystem since there was not any updated drivers availible. After rebooting normally into windows the laptop reinstalled fresh drivers.

After this I left the laptop running. With 24 hours of continuous uptime I was then able to hand the laptop back to the user confident the problem was resolved.

Pages

Archives

Search: