Windows Failover Cluster Live Migration Failures with Hyper-V 2012

After moving from one datacenter to another we started experiencing issues live migrating virtual machines from one host to another in our 2 node failover cluster.  The migration would instantly fail, and there would be no error other than:
Live migration of 'Virtual Machine VMNAME' failed.

If I did a quick migration it works, but live did not.  I started looking at the security logs of the hosts and noticed some intermittent errors:

An account failed to log on.

Subject:
Security ID: SYSTEM
Account Name: HYPERVHOSTCOMPUTER$
Account Domain: OURDOMAIN
Logon ID: 0x3E7

Logon Type: 8

Account For Which Logon Failed:
Security ID: NULL SID
Account Name: HYPERVHOSTCOMPUTER
Account Domain: OURDOMAIN.com

Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006D
Sub Status: 0xC000006A

Process Information:
Caller Process ID: 0xd30
Caller Process Name: C:\Windows\Cluster\rhs.exe

Network Information:
Workstation Name: HYPERVHOSTCOMPUTER
Source Network Address: -
Source Port: -

Detailed Authentication Information:
Logon Process: Advapi  
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

Then I noticed errors in the cluster itself, at the same times:

Cluster network name resource 'Cluster Name' failed registration of one or more associated DNS name(s) for the following reason:

The handle is invalid.

.

Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server.

I looked at a domain controller and noticed a lot of Audit Failures for that computer object.  I opened the computer object in ADSI Edit, and noticed that the last login was 11/23 (the day we moved), and the last password reset was 11/24, which is incredibly odd.  The last bad login attempt was a few minutes ago.  I'm not sure how, but I think a password reset may have been attempted while the domain controllers were unavailable.

How I fixed it:

1. Open Failover Cluster Manager
2. Navigate to Cluster Core Resource
3. Right click on the cluster network name and take it offline
4. Right click on the cluster name and navigate to more actions -> repair

A few seconds later the cluster was repaired, I turned the cluster name back on and live migrations work.

Mystery solved.

HTH!

Auto Provision Lync 2010 Users Via LDAP

We have tools that automatically create Active Directory Users, Exchange mailbox, and Microsoft Lync 2010 accounts to simplify the hiring process.  I figured somebody might find this information useful, so in order to provision a user for Lync 2010 automatically I use a slightly altered version of this (in perl, using Net::LDAP):

use Net::LDAP;


$ldap = Net::LDAP->new("ldap://domain.local", debug =>0) or die("Could not connect to LDAP server.");
my $mesg = $ldap->bind('myUserDN',
password => 'myUserPassword')  or die("Could not bind to LDAP server.");

 $mesg = $ldap->search( # perform a search

base   => "dc=domain,dc=local",

filter => "(&(samAccountName=$username))"

  );

$mesg->code && die $mesg->error;

#there should only be one result in here anyway

foreach $entry ($mesg->entries) { 

$userdn = $entry->dn;

}

$rtn = $ldap->modify($userdn, replace => { "msrtcsip-userenabled" => "TRUE"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-optionflags" => "449"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-primaryhomeserver" => "CN=Lc Services,CN=Microsoft,CN=1:1,CN=Pools,CN=RTC Service,CN=Microsoft,CN=System,DC=domain,DC=local"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-primaryuseraddress" => "sip:$email"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-line" => "tel:+$astextension;ext=$extension"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-deploymentlocator" => "SRV:"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-federationenabled" => "TRUE"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-internetaccessenabled" => "TRUE"});
$rtn = $ldap->modify($userdn, replace => { "msrtcsip-userpolicies" => [("21=7", "0=1434923910")]});

You may find it best to provision a user the way you like, open the user in ADSI Edit to see all the parameters, and adjust this accordingly.  

That said, this should set you off in the right direction for how to fully provision a Lync user automatically.

Automatically add Remote App connector via command prompt in Windows 8

We've been using the Install-RADCConnection.ps1 (I don't remember where I came across it) script to add the remoteapp connector in Windows 7, but it doesn't work properly in Windows 8.  I've found that a simple change needs to be made.  On lines 40, 42, and 44 change the reference to $connectionKey to be $Id and it works properly.

I was receiving this error when I ran it on Windows 8:


C:\Windows\system32>powershell -File \\server\scripts\Install-RADCConnection.ps1 "\\server\scripts\webfeed.wcx"
HKEY_CURRENT_USER\Software\Microsoft\Workspaces\Feeds\{50250AF5-D0E6-4ED5-BA69-D6CF96A7D55E}
Get-ItemProperty : Cannot find path 'C:\Windows\system32\HKEY_CURRENT_USER\Software\Microsoft\Workspaces\Feeds\{50250AF5-D0E6-4ED5-BA69-D6CF96A7D55E}'
because it does not exist.
At \\server\scripts\Install-RADCConnection.ps1:44 char:32
+         if ( ($connectionKey | Get-ItemProperty -Name URL).URL -eq $URL)
+                                ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Windows\syst...9-D6CF96A7D55
   E}:String) [Get-ItemProperty], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetIt
   emPropertyCommand

HKEY_CURRENT_USER\Software\Microsoft\Workspaces\Feeds\{50250AF5-D0E6-4ED5-BA69-D6CF96A7D55E}
Get-ItemProperty : Cannot find path 'C:\Windows\system32\HKEY_CURRENT_USER\Software\Microsoft\Workspaces\Feeds\{50250AF5-D0E6-4ED5-BA69-D6CF96A7D55E}'
because it does not exist.
At \\server\scripts\Install-RADCConnection.ps1:44 char:32
+         if ( ($connectionKey | Get-ItemProperty -Name URL).URL -eq $URL)
+                                ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Windows\syst...9-D6CF96A7D55
   E}:String) [Get-ItemProperty], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetIt
   emPropertyCommand


Connection setup failed.

Consult the event log for failure information:
(Applications and Services\Microsoft\Windows\RemoteApp and Desktop Connections).

Scripts Disabled

File \\server\scripts\Install-RADCConnection.ps1 cannot be loaded because

running scripts is disabled on this system. For more information, see

about_Execution_Policies at http://go.microsoft.com/fwlink/?LinkID=135170.

    + CategoryInfo          : SecurityError: (:) [], ParentContainsErrorRecord

   Exception

    + FullyQualifiedErrorId : UnauthorizedAccess

If you run in to that error, make sure you follow the steps in the link it provides to enable script execution.  We set ours at the GPO level.  http://technet.microsoft.com/library/hh847748.aspx

SQL Server 2012 Availability Group Replica Not Synchronizing

For some reason one of the databases in the availability group was stuck "Not Synchronizing" for a few days. A simple way to fix it was to simply pause & resume it using the SQL command found here: http://msdn.microsoft.com/en-us/library/ff877974.aspx

alter database <databasename> set hadr pause
alter database <databasename> set hadr resume

Synchronization immediately re-started and the database is now catching back up.  I'm not sure what happened, the event logs didn't show anything helpful but this did the trick.
:-\

Exchange Management Console 2010 Send on Behalf of Error

I was trying to add a user to send on behalf of somebody else, and when I tried to apply the settings I got the following error:

--------------------------------------------------------
Microsoft Exchange Error
--------------------------------------------------------
The following error(s) occurred while saving changes:

Set-Mailbox
Failed
Error:
Couldn't find object "domain.local/retired/staff/personsname". Please make sure that it was spelled correctly or specify a different object. Reason: The recipient domain.local/retired/staff/personsname isn't the expected type.


The solution was to open ADSI Edit, navigate to the user I was trying to grant the permissions for, find the publicDelegates setting, and remove the user being referenced.  Problem solved. :)

Panasonic KX-UT670 Executive SIP Phone Review

I saw the 7" Android Based Panasonic KX-UT670 VoIP SIP phone on display at Astricon in 2011 and instantly fell in love with it. On first impression the phone is well polished, well designed, and has a great feature set.

Features:

• Android 2.2 Based
• 6 SIP accounts
• 7 inch, color touch screen
• HD Voice (G.722)
• 100 entry phonebook
• 3-way conference call support
• Compatible with Asterisk and Broadsoft
• 2 x GbE ports, PoE
• Linux based open source operating system
• Full duplex speaker phone
• Electronic hook switch (Plantronics)
• Plug and Play configuration
• Cable concealing back
• Long handset cords
• Non-slip rubber feet
• Changeable angle stand
• Wall mountable

The first thing you notice with UT670 is it has a nice home screen providing easy access to your call history, contacts, voicemail, calendar, and of course the phone itself.  It's easy to handle calls using the built in Phone application and feels fairly natural.  The buttons on the right are flexible shortcuts.  If you press the arrow further down it expands to show a screen full of programmable shortcut buttons, where you can program things like speed dials or application shortcuts.

Full Shortcut Screen:

The settings page contains the normal android settings as well as a new Administrative Settings Menu:

The ethernet port settings allow you to configure VLAN tagging, and set port speeds.  IP configuration is done outside of the admin menu.
Applications can be installed via SD Card (side loading), or via the web.  Simply copy the APK to an sd card or install an application from the web browser (you must open this via admin mode, the normal browser will not allow APK's to be installed).  *Note that you need an SD card in the phone in order to install an application, but once the application is installed I've found that I can remove the SD card without any problems.

Additionally, you can add your Google account via the normal Accounts & Sync menu to sync your contacts and calendar for you, but unfortunately it seems to download a contact for every person you've ever corresponded with rather than the contacts in your actual contact list, making it difficult to use..

Configuration of the phone can be done automatically using provisioning files, or manually via the web interface of the phone (username admin, password adminpass).  The web interface doesn't expose all of the potential options, to get access to everything you need to check out the admin guide and use a provisioning file.  Firmware updates aren't terribly difficult to do, I found I had to specify the version of the firmware, and a link to the new firmware version in a provisioning file in order to get the phone to update because the web interface options did not seem to work.

The KX-UT670 has a good speakerphone, I use it almost exclusively, rarely ever picking up the handset.  I haven't had any issues hearing the other person, or with the other person being able to hear.  I've only used it in a private office but it has worked great so far.

One thing the phone is missing is built in bluetooth support, which the Aastra 6739i and the CloudTC Glass phones both have.  If you really want to use a bluetooth headset supposedly the UT670 can support bluetooth via the USB port, but I have not tested this.

In October I spoke to the Panasonic Engineers and told them what I was trying to do and they said they said they would be adding much of the functionality I was looking for in the second release of their API.  I received that API and was easily able to modify my application to control call flow inside the Panasonic emulator.

Here is a screenshot of the application I've built, an intuitive attendant / reception console.  It is a great replacement for the old sidecars like the Aastra 560M or the Polycom IP 650 Sidecar Expansion Module that take up lots of desk space, provide less functionality, and cost a lot more.  If you are interested in this you can find out more at http://www.lighttouchsoft.com.

Selecting a person's name provides a list with their extension and cellphone, providing the ability to easily route calls to them.

The Good:

• The phone is gorgeous and the handset has a great feel to it.  I like the hard buttons at the bottom of the screen, I wasn't convinced at first but they definitely make usability easier than the CloudTC phone.
• The API is fairly easy to use and work with, although the documentation isn't very good so you may have to do a bit of trial/error to figure out how to do some things.
• It works quite well.
• At $400 it's relatively cheap for what you get.

The Bad:

• The API has a couple shortfalls, and Panasonic has responded that they designed it that way.
• When you place a call on hold a new line is opened up so you can make another call.  This is great unless you are putting them on hold to research something, talk to somebody that walked up to you, or any other reason you would put a caller on hold but not want to open another line.
• If you hang up the new call the phone starts beeping in a very, very obnoxious manner.  Panasonic has said this is so the user doesn't forget a call is on hold.  I would prefer the API let me, the developer, control the user interaction with the phone.
• The base lets you tip the phone at either a very steep angle or a very flat angle with no in between option.  Neither one of the settings is very convenient for my desk space so I've put a postit pad under the back of it to tip it up a little more and create an in between option.
• According to Panasonic you may not use these characters in your passwords: " & ' < >.  If one of these characters included in the config file the phone will ignore the entire config file, and if you try and enter them in the web interface the page reloads with the old password.  It took quite a while to figure out this was the problem, but fortunately I've seen this happen with the Panasonic TGP500 so I wasn't flying completely blind.  It would be nice if they included this in the documentation somewhere.  
• There are a few things like firmware updating, passwords, and a couple API tweaks that I would like to see made, but the phone was just recently released so I expect that will happen over the next few months.

In case you are trying to figure out a quick and easy basic provisioning file, here's mine to get you started.

# Panasonic SIP Phone Standard Format File
CFG_CYCLIC="Y"
CFG_CYCLIC_INTVL="1200"
HTTPD_PORTOPEN_AUTO="Y"
NTP_ADDR="domaincontroller.domain.com"
## SIP Settings
PHONE_NUMBER_1="1593"
SIP_RGSTR_ADDR_1="pbx.domain.com"
SIP_PRXY_ADDR_1="pbx.domain.com"
SIP_AUTHID_1="1593"
SIP_PASS_1="password"
DISPLAY_NAME_1="Andrew Parisio"
SIP_P_PREFERRED_ID_1="Y"
ADD_EXPIRES_HEADER_1="Y"
ADD_TRANSPORT_UDP="Y"

P.S. Here's how you do a factory reset of the phone:

To factory reset the phone through web interface. Log on to the phone using username: customerserviceid

password: cspass

Data Protection Manager 2010 February 2012 Update Crash

Update: Microsoft has been able to reproduce the error and believes this is has been introduced in the latest February 2012 update.
Update 2: It has since been confirmed as a bug, with a hotfix coming.  In the meantime if you run in to this problem they should be able to fix it for you.  The bug is triggered when many events are written to the log at the same time, such as a hardware failure causing all active jobs to fail.


We use Microsoft Data Protection Manager (DPM) 2010 to protect our environment, including our Hyper-V failover clusters which use Cluster Shared Volumes (CSV).  Last Wednesday we installed the February update, and then on Friday one of our McData 4700 4GB FC switches had a port failure which crashed the entire SAN fabric (including the entirely separate, second path which was in no way tied to the primary path).  During this outage, which should never have happened in the first place, our DPM server corrupted itself.  The DPM server is completely separate from the Hyper-V Cluster, the only thing they share is the same network; therefore there is no reason for DPM to have crashed like this.

Now, whenever we attempt to start DPM we get this error in the event viewer (and the DPM service fails to start):

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.


If the event originated on another computer, the display information had to be saved with the event.


The following information was included with the event: 


An unexpected error caused a failure for process 'msdpm'.  Restart the DPM process 'msdpm'.


Problem Details:
<FatalServiceError><__System><ID>19</ID><Seq>0</Seq><TimeCreated>4/23/2012 6:07:09 PM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>NullReferenceException</ExceptionType><ExceptionMessage>Object reference not set to an instance of an object.</ExceptionMessage><ExceptionDetails>System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.BackupEventIntegration.WriteBackupEvent(BackupEventEntry backupEventEntry)
   at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.BackupEventIntegration.WriteNonLoggedBackupEntries()
   at Microsoft.Internal.EnterpriseStorage.Dls.Prm.PRMHealthProvider.Initialize()
   at Microsoft.Internal.EnterpriseStorage.Dls.JobManager.JobManager.Initialize()
   at Microsoft.Internal.EnterpriseStorage.Dls.JobManager.JobManager.InitializeIfNecessary(Object state)
   at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</ExceptionDetails></FatalServiceError>




the message resource is present but the message is not found in the string/message table

I opened a support ticket with Microsoft and within half an hour had a call from a tech.  After an hour of trying different things and digging around he asked for a copy of all of our logs, but the diagnostic logging tool they gave me wouldn't work properly because DPM had created 110GB of error log files in 3 days... whoops.  After deleting the repetitive crash logs (9,000 of them, around 12MB each) he got the upload and was able to look at the data.  He has since come to the same conclusion I have, that the crash was indeed a problem in DPM (no, it should not corrupt itself when the sources it is protecting go offline).  He has sent our (corrupt) DPM SQL DB to the "EE" team to look at it, but who knows how long it will take for them to figure out what went wrong and create a fix for us.  In order to get back online we uninstalled DPM, reinstalled, upgraded to QFE 2, and restored from the backup database from Wednesday when the February update was installed.  We then upgraded to the February update again because all of the protection agents needed it, and have set up a task to 

The good news is you don't need the DPM management console or service to be able to restore data.  In the event of a major SAN failure it would have been possible to recover our data using manual tools (with the help of microsoft support).  This is still very unnerving, and a prime example of why you should have good backups of the SQL database on your DPM server.