W3WP Process High CPU and RAM utilization on Exchange 2010 CAS

Last weekend I updated Exchange to SP2 and UR3 and have been keeping an eye on the server occacionally to make sure its running normal.  Today I noticed in the Exchange Server Performance Monitor that there was a lot of spiking and more than normal activity.  The extra activity was for the processing, so I went into the task manager to take a look at what process was using it all, as we normally run between 5-10% utilization on this server, and it was spiking up to 60% at times and consuming over 1GB of memory.

I checked out the Event Viewer and didn’t see any errors.  There were quite a few warnings from ActiveSync for an iPad device for user dsims, could this be linked to the IIS process issue?

Next I downloaded Process Explorer and located the troubled process and gathered some more info.  The IIS worker process was linked back to MSExchangeSyncAppPool.  I went into the Performance Graph and noticed something odd, there was a lot of CPU usage and IO and then it stopped, and started again.

At this point, I did a google search for ‘MSExchangeSyncAppPool high memory usage’ and many pages linked back to an issue with iOS devices.  Since I know who has the iPad in our company, I messaged them to see if they could take it offline for a while to see if that resolved the issue, which it did.

My concern is why was this process so heavily used?  Was it a bug, was the user downloading all of their mail, what caused the issue?

HowTo View Exchange Services Health

Last weekend I installed SP2 and UR3 onto our Exchange 2010 CAS and Hub Transport servers, along with the mailbox server.  Mail flow seemed fine afterwards and I was able to send and receive emails, but sometime over the weekend two services stopped.  I started the services Monday morning before everyone got it, but some automated reports ended up not running over the weekend.

Solution 1: I put a powershell script in place that I found online that monitors the Exchange Servers and emails me daily at 8PM through a scheduled task.  I will eventually be modifying it to show the hard drive space of all the partitions.

Solution 2:  There is a command you can run in the Exchange Management Shell that will show you the status of all the Exchange Services on each server, and show which are running, and which are not.  This command is Test-ServicesHealth.

Windows Server 2012 Licensing Changes

-Two versions of Server 2012 – Standard and Datacenter
-One 2012 Datacenter license is good for 2 physical CPU
-Can not licence both Standard and Datacenter on same physical server
-SA with 2008 R2 Datacenter will be converted at a 2:1 ratio for 2012 Datacenter
-Will need to purchase 2012 CAL’s
-Standard License – $882
-Datacenter License –  $4,809


Updating HP t5545 Thin Client

I have a HP t5545 thin client that I got for testing View 5 on.  Our initial plan was to re-use our Dell PC’s with Windows Thin PC to keep from buying new hardware, but I still wanted to test thin clients.  So I picked up this thin client for $300 and its been sitting on the side having only been used once or twice.  I finally have some time now that I have the View infrastructure up to begin testing with the thin client some more.

The OS HP Thin Pro is quite out of date
BIOS Version 786R5 v2.02
BIOS Release Date 6/10/2010
OS Build ID T5X31012
OS Kernel Version 2.6.26-2-686

After downloading the above file, I ran it on my Windows 7 workstation, and was prompted to create an ISO image, USB format, or Deployment.  Since the thin client doesnt come with a optical drive, I chose USB.   I then selected the M: which was a 8GB empty flash drive, and formatted the USB.









Once this finished, I put the USB into the thin client and rebooted it.  It prompted me with a blue screen it was going to format the onboard thin client Flash disk and erase all data, and update all the software.  Click entered Y.  The process began.  Took about 5 minutes.

While I was waiting for this to finish…. how do we update this on 200 thin clients effectively?

Removed the USB and rebooted the thin client, Clicked Launch Easy Setup Utility.
The first tab in the Setup Utility was updates, I applied all the updates available.

After updating, here are the new versions
OS Build ID T5X32112
OS Kernel Version 2.6.27-17-generic

VMware View 5 Setup and Configuration with vShield

I have slowly been working on getting a 10-user pilot going for View in our 10 computer lab.  I have been working on other projects quite a bit, and haven’t had a whole lot of time, but finally had about 8 hours over 2 days to get it setup and going.

The hardware I am using for this is a HP DL360 G5 with 32GB RAM, and 6 146GB 15k drives with dual quad core 2.5ghz Xeons.  In my rough calculations I did a few months ago, i figured this would be perfect size to host the two or three VM’s for the vCenter, Composer, Connection, and vShield servers, along with 10 virtual desktops.

Connection Server = 2 vCPU, 4GB
vCenter and Composer =  2vCPU, 4GB
vShield Manager =  1vCPU, 3GB
(10) Windows 7 Virtual Desktops = 2 vCPU, 2GB
This totals out to 31GB of the total 32GB capacity

I ended up purchasing a View 5 Premier Bundle Starter Kit ($2,500) which includes most everything you need to get a pilot going for 10 virtual desktops, including the server side VMware software.  We already had the Windows 7 Enterprise licensing, and the Server 2008 R2 licensing so nothing was needed for Microsoft licensing.  In order to take advantage of vShield we were going to integrate it into TrendMicro OfficeScan AV which is already rolled out the company for our main AV solution.


Our PC’s are replaced every 6 years, and in order to justify VDI for the computer lab it was pretty simple.  Replace 10 PC’s at $1000 each for $10,000, or roll out VDI using existing PC hardware and existing spare server, for $2,500, for cost savings of $7,500.  The overall long-term goal is to roll-out VDI to the entire company, so a pilot is definitely needed to show proof of concept.

  1. Install ESXi onto DL380 G5
  2. Install vCenter in a virtual machine with SQL Express DB
  3. Patch ESXi host
  4. Setup additional DB in SQL Express for View Composer
  5. Install View Composer and attach to DB
  6. Install and Configure View Connection Server
  7. Create Windows 7 Enterprise x64 base image
  8. Create 10 linked clones from base image
  9. Install and Configure vShield Manager ova template
  10. ThinApp applications and rollout to Desktop Pools


vShield Setup with OfficeScan
Download Deep Security Virtual Appliance 8.0
Download  FilterDriver-ESX-5.0
(15) TrendMicro OfficeScan VDI Plug-in License $250

ThinClient vs Reusing Thick Clients
Our computer lab consists of Dell Vostro PC’s in which we will be installing Windows Thin PC OS onto it and booting into in order to boot the View Client.  We looked into other choices such as ThinStation, Windows PE, or a basic linux install, the problem with that is the maintenance and learning curve and troubleshooting of learning something new.

After installing TPC onto a Dell Vostro, I logged in and installed VMware Tools.  At this time, its not going to be joined to the domain.  As I was installing this, some thoughts started rolling through my head:

  1. How do we manage windows updates on these without being joined to the domain?
  2.  How is AV going to be handled on the TPCs?
  3. How can we lock down TPC further so only troubleshooting tools and View Client work?
  4. Can we discard any changes made at reboot? Write Filters
  5. How do we manage the images and TPC?  SCCM supports TPC, and WSUS also
  6. What does ThinPC allow to be run? Remote Desktop Clients, Management, Security, Media Player applications only
  7. Do we need to keep an active Software Assurance agreement to keep TPC?
  8. Applocker, block IE, Media Player, etc

vMA and Patching Single ESXi5 host

I am in the process of rolling out a 10-user pilot for our computer lab on an older DL360G5 we have that isn’t getting used for anything at the moment.  Since I am keeping this host separate from our server infrastructure, I dont have the luxury of using update manager and moving vCenter and the other VM’s off to install updates and reboot the host on demand.

I wasn’t sure how to accomplish this, but found a link here that looked like it was what i was looking for.

In order to patch an ESXi5 host without using Update Manager I went to VMware’s patch download website and downloaded the latest set of patches.

In order to apply these patches I need vSphere Management Assistant also which is supplied by VMware as an OVF file.  I downloaded and deployed the OVF into VMware Workstation on my Desktop.  Upon starting the first time, I received the following error.

I haven’t seen this error before, or know anything about IP Pools yet, so I looked up the error and found this link which I used to resolve the issue and would allow me to boot up the vMA VM. I went through the process of setting up vMA which is pretty self explanatory.

-Next I logged into the vMA using vi-admin and my password I set.
-Next I uploaded the  .zip update file I downloaded from the VMware website onto the local datastore
– Went through the rest of this document to apply the patches to the ESXi host

ActiveSync Remote Wipe caveat

Today I had to remotely wipe a users phone that was setup to our Exchange 2010 server through ActiveSync.  I haven’t done this yet, since we are migrating from BlackBerry’s and usually just do a Remote Wipe on the BES server, so I figured I would try it out with ActiveSync and get a documented process and KB article created.

Ran the following commands in the Exchange Management Shell

Get-ActiveSyncDeviceStatistics -Mailbox bstollfus | fl Identity

Identity: internal.domain.com/Information_Systems/SystemsAdmins/Users/Brad Stollfus/ExchangeActiveSyncDevices/SAMSUNGPHD710SAMSUNGA00000F78349FA

Clear-ActiveSyncDevice -Identity internal.domain.com/Information_Systems/SystemsAdmins/Users/Brad Stollfus/ExchangeActiveSyncDevices/SAMSUNGPHD710SAMSUNGA00000F78349FA -NotificationEmailAddress “bstollfus@internal.domain.com

Initially the wipe didnt work so I rebooted the phone which didnt resolve the issue so I started to look into why.  It turns out that because the way ActiveSync works with the remote wipe flag, if the user is disabled in Active Directory, the phone is unable to authenticate (obviously) and the remote wipe flag can not get received by the phone if it is unable to authenticate.

Here is an article that goes into the details a little bit more.

Advantages of VAAI and Enabling VAAI on Lefthand P4500 G2 SAN with vSphere 5

In preperation for taking the VCP5 exam I have been doing a lot of reading on vSphere 5.  One of the topics was VAAI which is vSphere Storage API’s for Array Integration(VAAI). After some Google searching, it appears that LeftHand SAN’s do support VAAI.  So that is great news, since it can dramatically increase performance.  What VAAI does it offload some of the storage tasks from the ESXi hosts, onto the storage array itself.  Below are some more specific details of what it does and how it improves performance.

Array Integration allows for Hardware-Assisted Locking which locks on a per sector basis instead of locking the entire LUN.  This can have a substantial increase in performance when a lot of changes in metadata occur, such as when many VM’s are powered on at once.

Hardware-Accelerated Full Copy allows for the storage itself to make entire copies on its own, without having to send any read/write requests through an ESXi host.  Events such as cloning VM’s or deploying new VM’s from templates have a significant reduction in storage traffic between the ESXi host and the array.

Hardware-Accelerated Block Zeroing allows storage arrays to zero out blocks very quickly and speeds up the process of creating new VM’s and formatting virtual disks.

vSphere5 is also thin provisioning aware, and when coupled with VAAI allows you to reclaim dead space and give you advanced warning when approaching out of space conditions.

VAAI Whiteboard
VAAI Performance Info

By default, VAAI is enabled and supported with ESXi 5 so nothing had to be done in vSphere 5 or on the P4500 SAN.

Allowing Users to Manage Distribution Lists in Exchange 2010

Apparently by default, when you go into a Distribution Group and go to the Group Information Tab and assign users to the Managed by: area, this is not enough to allow them to make changes.  After some investigating I found the correct PowerShell commands to run to allow them to modify groups they are managing, and not allow them to create or delete distribution groups.

New-ManagementRole -Name OwnerDistributionGroups -Parent MyDistributionGroups

Remove-ManagementRoleEntry OwnerDistributionGroupsNew-DistributionGroup -Confirm:$false

Remove-ManagementRoleEntry OwnerDistributionGroupsRemove-DistributionGroup -Confirm:$false

New-ManagementRoleAssignment -Role OwnerDistributionGroups -Policy “Default Role Assignment Policy”


Our trainer was going to sit down with the users that were managers of these groups.  She requested a list of the distribution lists, along with who was able to manager the lists.  Ran the following command to get the ManagedBy info:

Get-DistributionGroup | fl Name,ManagedBy > C:distro-managedby.txt

Removing Emails from Exchange Database with Specific Subject using PowerShell

As an Exchange administration we often find ourselves dealing with panicked users who just sent an email they shouldn’t have.  If you don’t deal with it enough, you may not have the PowerShell command memorized, which is why its a good idea to document it so its easily accessible.  This command works well if the subject is very specific, if its generic, you may end up deleting emails you did not intend to.

Get-Mailbox –resultsize unlimited | Search-Mailbox –SearchQuery “Subject:Spiff Results as of 2-27-12” –DeleteContent