relocating iSCSI volume with db/logs to a new server

We have an Exchange 2007 Mailbox server running on Server 2003. We want to build a Server 2008 box, and attach the Exchange iSCSI volume to the new server.

https://technet.microsoft.com/nl-nl/library/bb123954(v=exchg.80).aspx

As with previous versions of Microsoft Exchange, an upgrade of the operating system for an Exchange server results in the updating of the value for OS Version in the database header. This update triggers the rebuilding of internal database indexes. When using database portability to move a database from a Mailbox server running Windows Server 2003 to a Mailbox server running Windows Server 2008, the Extensible Storage Engine (ESE) detects the operating system upgrade and takes the following actions:

  • During the first database mount operation, all secondary indexes are discarded. A secondary index is used to provide a specific view of the mailbox data (for example, when messages in a mail folder are sorted using Outlook in Online mode). The database will not be mounted and available to clients until this initial operation is complete. The amount of time to complete the operation is largely dependent on the size of the database. The larger the database is, the longer the mount operation will take.
  • Secondary indexes will be rebuilt on-demand as Outlook users sort their views in Online mode. In environments with large or extremely large databases, the on-demand rebuilding of indexes will initially result in high processor and disk utilization.

Unmount databases on old Exchange MBX server
Stop Exchange Services on old MBX server
Disconnect iSCSI volume from old MBX server
Connect iSCSI volume to new MBX server
Mount iSCSI drive in Windows on new MBX server
Create Storage Groups and point to existing DB/Logs on iSCSI volume
Mount databases
Wait for indexing to take place before database remounts
Run PowerShell command to point mailboxes from old MBX to new MBX

Get-Mailbox -database “EXMBX1\CORP Storage Group\Mailbox Database” | Move-Mailbox -TargetDatabase “EXMBX3\CORP Storage Group\Mailbox Database”| -ConfigurationOnly

4.4.2 Connection Dropped Exchange 2007

Had an issue with a couple users trying to send to 1 specific domain that just seemed to have stopped working.  Exchange queue on our Edge server was showing the following error: 4.4.2 Connection Dropped

After several hours of research, and determining it was nothing on our side, I contacted the other sides IT team who reached out to rackspace who was hosting their email.

Logs on Edge on our side showed the following

C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\Connectivity

2016-07-01T00:23:29.590Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX1

2016-07-01T00:23:30.043Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX2

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,>,Failed connection to XXX.XXX.XXX.XX3 (0000274C)

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,-,Retry

Why was it establishing a connection, but moving onto the next MX record still?

Ran MX toolbox

Ran Microsoft Exchange Connectivity Analyzer which showed it wasnt able to connect to the 3rd MX record

Questioned what the 3rd MX record was, and it was a stale on-prem email record that was never removed.  They updated DNS to remove this.  Let it propagate the internet

Ran a ipconfig /flushdns on Edge server

Cleared the messages on the Edge server in this domains queue, Suspended the Queue, Resumed it to clear it out, sent another email and it went through successfully.

Cause: Stale MX record on recipients side

Conclusion: Only explanation I can come up with is that the first 2 MX records for rackspace were unavailable and it tried and hung onto the 3rd stale MX record and kept trying to use that.

I was going through an Exchange Edge server queue and noticed 100+ emails with Last Error “A local loop was detected”

LocalLoop

Did a little investigating and found the primary smtp namespace was set as an internal relay type in the Accepted Domains

AcceptedDomains

Changed these to Authoratative to resolve the local loop issue

https://social.technet.microsoft.com/Forums/exchange/en-US/8e77f4b0-5e34-42b1-a68d-84c894c5976b/exchange-2007-last-error-a-local-loop-was-detected?forum=exchangesvrsecuremessaginglegacy

Server 2003 stuck at startup splash screen HyperV VM

Had a 2003 Server VM go down because of a network/SAN connectivity issue.  Upon starting the VM back up, it got stuck at the Windows splash screen indefinitely.

2003splash

Tried booting into safe mode and got stuck on ACPITABL.DAT

acpitabl.dat

Connected VM to 2003 Server ISO and booted off that
Hit R for recovery
Sat at Examining disk for 10-15 minutes

mbx3 not booting

Finally got in and ran a checkdisk

checkdisk1

SOLUTION: Ended up being an iSCSI disk that was connected to the VM in Hyper-V that was causing it to hang.  When the iSCSI disk was unmounted, server booted up fine and it was remounted.

Restore Exchange 2007 Databases from Tape to new Mailbox Server

We had 6TB of email databases go offline from a double disk RAID5 failure.

Built new Exchange 2007 Mailbox Server on 2008 R2 as we took this opportunity to get the failed mailbox server off 2003 server.

Restored edb and log files from tape to new mailbox server MBX4.
D:\DB1\Mailbox Database.edb
D:\DB1\Logs

Replay Logs using eseutil using the following command
Go to D:\DB1\Logs directory
E02 is the checkpoint file, this is different, look at the first 3 of the logs prefix to determine this
eseutil.exe /R E02 /I /DD:\DB1\ /SD:\
This process can take 5-60 minutes depending on how many log files there are.

eseutil

Once this completes create a new storage group and point it to the Database Folder, and the Database Logs folder. This will take a bit to create, so be patient.

StorageGroup-2

Rename database edb file
Rename Mailbox Database.edb to Mailbox Database.edb.oldCreate a new Database in the storage group you just created called Mailbox Database.edb, DO NOT mount the database.
Rename the Mailbox Database.edb.old back to original, remove .old

Go into the database properties and adjust the quota’s and make sure you check ‘This database can be overwritten by a restore’

DBProperties

Mount the Database

Run the following powershell command to re-home the mailboxes from the old mailbox server to the new mailbox server

[PS] C:\Windows\system32>Get-Mailbox -Database “MBX1\Storage Group
\Mailbox Database” | where {$_.ObjectClass -NotMatch ‘(SystemAttendantMailbox|ExO
leDbSystemMailbox)’}| Move-Mailbox -ConfigurationOnly -TargetDatabase “EXMBX4\Storage Group\Mailbox Database”

Run the following command to show all the mailboxes in the database and export to a txt file so you know who’s mailbox is back online

[PS] C:\Windows\system32>Get-Mailbox -Database “Storage Group\Mailbox Data
base” > C:\DB1.txt

Outlook Connection Status

What are good numbers to have in  Outlook Connection Status?
These numbers are for cached mode.  Online mode you will want 50ms or lower otherwise Outlook will constantly be hanging to grab the information from the server.

  • Max Avg Proc Time (Exchange RPC Latency) = 25ms
  • Max Avg Resp Time (Exchange RPC Latency + Network Latency) = 325ms

OutlookConnectionStatus

To get to the Outlook Connection Status:

  1. Launch Outlook
  2. Hold Ctrl and left click on the Outlook system tray icon
  3. Select Connection Status…

Random connection drops HyperV Proliant Gen8 server

We have an Exchange environment on an HP Proliant DL380p Gen8.  The VM’s came from old physical servers that were relocated to another site, ever since they were virtualized and put onto new hardware, there have been random network issues.  We could ping between all Exchange servers one minute, next minute, cant ping anything.

Found an article on Microsoft site about a bug in Broadcom Chipset with HyperV and Server 2012 R2.  Looked up the HP FlexLOM card, and sure enough, it uses BCM5719 chipset, one of the affected chipsets.  Also the driver we are using is 16.4 and the issue was resolved in 16.8.

https://support.microsoft.com/en-us/kb/2986895

https://www.broadcom.com/docs/features/netxtreme_ethernet_hp_datasheet.pdf

HP-FlexLOM-Broadcom

Exchange 2007 Mailbox Server VM bluescreen

Had an issue over the weekend, one of our mailbox servers kept bluescreening.  This particular VM has the C: and D: on  storage iSCSI SAN.

Tried running checkdisk on C: and it bluescreened as well.  HyperV manager shows the error “Cannot connect to virtual machine configuration storage”

The D: drive is showing uninitialized in disk management, and gives an error when trying to initialize.

Started getting “Windows – Delayed Write Failed” messages.  Compression is not turned on on the HyperV servers local storage.

The iSCSI connections are provided by HP Lefthand SAN, 2 node cluster.  One of the SAN nodes went down yesterday.

HP Lefthand Support Phone Number
1-800-633-3600
shortcut –
Lefthand Solutions

putty into SAN node : port 16022

2 Lefthand SAN nodes
Each has a single 100mbit link….
When we brought SAN2 back online state, issue was resolved
My assumption is that the SAN couldn’t replicate fast enough with the load they have with Exchange on them, plus replicate the changes to the other SAN node, so they were out of sync when the 1 SAN node went down thus causing issues.

mbx3-8

mbx3 mbx3-2 mbx3-4 mbx3-5 mbx3-6 mbx3-7

mbx3-8

Stollfus Tech Blog

WordPress Appliance - Powered by TurnKey Linux