Category Archives: Exchange 2007

EternalBlue Exploit used to Deliver Remote Access Trojans

EternalBlue Exploit Actively Used to Deliver Remote Access Trojans

On May 16th we started experiencing mail flow issues on our unsupported Exchange 2007 environment hosted on Server 2003 OS’s.  Mail was coming in over SMTP and being delivered to users as normal, however internal emails, and outgoing emails appeared to be going into $null.  I could not see them in any Exchange logs, there was no trace of what was happening to them.

After some investigating, another engineer found two ports were blocked, 135 and 445 or RPC and MS Directory Services.  The Windows firewall on all machines is disabled by default so we double checked that and moved on.  First stop, our antivirus, we disabled and then completely uninstalled on what servers still had it running, wait how was AV missing on these servers?  Next we checked our other fancy AI machine learning super advanced security application, Cylance, that was missing as well… hmm…

The workday had ended and myself and the other engineer were at home for the evening, i was VPN’ed in, determined to find what was blocking this traffic.  I had seen a few event logs referencing IPsec and a GUID but I hadnt thought much on those, i did a quick google search on IPsec rules and SErver 2003, loaded up the IPsec Management Console and noticed there was a new rule created, last modified a few hours earlier, the rule was simply called ‘win’.  Upon entering the rule I noticed there was a deny list, and sure enough, ports 135 and 445 were listed, unassigned the win IPsec rule, and mailflow started up again.  I checked the 3 other 2003 servers, same rule, unassigned, mailflow issues resolved.

Also during investigation, in event logs, i found an MSI trying to install when the other admins were logging in.  js.mykings.top was the name, and I found this in registry, scheduled tasks, and startup.

New users not showing up in OAB but show in GAL in OWA

Event ID 9320
OALGen could not generate full details for some entries in the offline address list for address

Get-OfflineAddressBook – Identity “Default OAB” | fl
GUID: 2e91c924-5590-4013-94a2-0dc08fe9285e

I checked the OAB folder on the Generation Server, and the 2 distribution servers, and noticed one of the distribution servers had files that were not modified with todays date, but the generation server did, as did one of the distribution servers.  So it appears to be a replication issue.

D:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB\2e91c924-5590-4013-94a2-0dc08fe9285e

Our 2 HUB servers are set as the OAB Distribution points, but is showing files last modified 10/11/2016 while the other is today, 10/12/2016.  So there seems to be a replication issue between them.

I ran Update-FileDistrubtionService – Identity EXCH01 -Type OAB
This ran about 15 minutes and didn’t finish

On the server without the updated OAB lzx files I restarted the Microsoft Exchange File Distribution service

The lzx files last modified date then updated to today’s date.

From my Outlook i downloaded a new copy of the OAB from Send/Receive and tested by creating a new email and confirming the user missing was now available

OAB/GAL not updating with new users

-Create new OAB on generating server
-Navigate to C:\Program Files\Microsoft\Exchange Server\ExchangeOAB and ensure there is a folder there -You may have to restart the Microsoft Exchange System Attendant service
-Go to Organization Configuration/Mailbox and Offline Address Book tab, make sure you are doing Web-Based distribution
-Navigate to your distribution servers to C:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB and make sure the folder is there as well.  If it is not, restart the Microsoft Exchange File Distribution service and wait a bit.
-Go into IIS on the distribution servers, Application Pools, Recycle MSExchangeAutodiscoverAppPool
-In Outlook do an Test E-mail AutoConfiguration and ensure the OAB UID matches the folder name in the above directories

Unable to create storage group to many log files

As part of disaster recovery, I was trying to restore a database and 45,000 associated transaction logs.  I was getting the following error when trying to create a new storage group pointing to the restored data.
Error:
Failed to connect to the target server “MBX-05”. The exception message is “WMI exception occured on server ‘MBX-05’: Call cancelled “.

Exchange Management Shell command attempted:
new-StorageGroup -Server ‘MBX-05’ -Name ‘ Storage Group’ -LogFolderPath ‘F:\ExchangeGroups\StorageGroup\Logs’ -SystemFolderPath ‘F:\ExchangeGroups\StorageGroup\MailboxDatabase’

OST sync with mailbox after restore

If we restore Exchange 2007 data from 2 weeks ago and bring it back online, will Outlook clients sync the previous 2 weeks of email from the OST back to the Exchange mailbox after its restored from backup?

I seem to see split opinions on this, and no documentation to support either claim if it will or will not.

We just tested.  Outlook did not sync the previous 2 weeks of emails as I had thought back with Exchange.  However those 2 weeks of emails did stay put in the users Outlook so far…

relocating iSCSI volume with db/logs to a new server

We have an Exchange 2007 Mailbox server running on Server 2003. We want to build a Server 2008 box, and attach the Exchange iSCSI volume to the new server.

https://technet.microsoft.com/nl-nl/library/bb123954(v=exchg.80).aspx

As with previous versions of Microsoft Exchange, an upgrade of the operating system for an Exchange server results in the updating of the value for OS Version in the database header. This update triggers the rebuilding of internal database indexes. When using database portability to move a database from a Mailbox server running Windows Server 2003 to a Mailbox server running Windows Server 2008, the Extensible Storage Engine (ESE) detects the operating system upgrade and takes the following actions:

  • During the first database mount operation, all secondary indexes are discarded. A secondary index is used to provide a specific view of the mailbox data (for example, when messages in a mail folder are sorted using Outlook in Online mode). The database will not be mounted and available to clients until this initial operation is complete. The amount of time to complete the operation is largely dependent on the size of the database. The larger the database is, the longer the mount operation will take.
  • Secondary indexes will be rebuilt on-demand as Outlook users sort their views in Online mode. In environments with large or extremely large databases, the on-demand rebuilding of indexes will initially result in high processor and disk utilization.

Unmount databases on old Exchange MBX server
Stop Exchange Services on old MBX server
Disconnect iSCSI volume from old MBX server
Connect iSCSI volume to new MBX server
Mount iSCSI drive in Windows on new MBX server
Create Storage Groups and point to existing DB/Logs on iSCSI volume
Mount databases
Wait for indexing to take place before database remounts
Run PowerShell command to point mailboxes from old MBX to new MBX

Get-Mailbox -database “EXMBX1\CORP Storage Group\Mailbox Database” | Move-Mailbox -TargetDatabase “EXMBX3\CORP Storage Group\Mailbox Database”| -ConfigurationOnly

4.4.2 Connection Dropped Exchange 2007

Had an issue with a couple users trying to send to 1 specific domain that just seemed to have stopped working.  Exchange queue on our Edge server was showing the following error: 4.4.2 Connection Dropped

After several hours of research, and determining it was nothing on our side, I contacted the other sides IT team who reached out to rackspace who was hosting their email.

Logs on Edge on our side showed the following

C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\Connectivity

2016-07-01T00:23:29.590Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX1

2016-07-01T00:23:30.043Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX2

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,>,Failed connection to XXX.XXX.XXX.XX3 (0000274C)

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,-,Retry

Why was it establishing a connection, but moving onto the next MX record still?

Ran MX toolbox

Ran Microsoft Exchange Connectivity Analyzer which showed it wasnt able to connect to the 3rd MX record

Questioned what the 3rd MX record was, and it was a stale on-prem email record that was never removed.  They updated DNS to remove this.  Let it propagate the internet

Ran a ipconfig /flushdns on Edge server

Cleared the messages on the Edge server in this domains queue, Suspended the Queue, Resumed it to clear it out, sent another email and it went through successfully.

Cause: Stale MX record on recipients side

Conclusion: Only explanation I can come up with is that the first 2 MX records for rackspace were unavailable and it tried and hung onto the 3rd stale MX record and kept trying to use that.

I was going through an Exchange Edge server queue and noticed 100+ emails with Last Error “A local loop was detected”

LocalLoop

Did a little investigating and found the primary smtp namespace was set as an internal relay type in the Accepted Domains

AcceptedDomains

Changed these to Authoratative to resolve the local loop issue

https://social.technet.microsoft.com/Forums/exchange/en-US/8e77f4b0-5e34-42b1-a68d-84c894c5976b/exchange-2007-last-error-a-local-loop-was-detected?forum=exchangesvrsecuremessaginglegacy

Restore Exchange 2007 Databases from Tape to new Mailbox Server

We had 6TB of email databases go offline from a double disk RAID5 failure.

Built new Exchange 2007 Mailbox Server on 2008 R2 as we took this opportunity to get the failed mailbox server off 2003 server.

Restored edb and log files from tape to new mailbox server MBX4.
D:\DB1\Mailbox Database.edb
D:\DB1\Logs

Replay Logs using eseutil using the following command
Go to D:\DB1\Logs directory
E02 is the checkpoint file, this is different, look at the first 3 of the logs prefix to determine this
eseutil.exe /R E02 /I /DD:\DB1\ /SD:\
This process can take 5-60 minutes depending on how many log files there are.

eseutil

Once this completes create a new storage group and point it to the Database Folder, and the Database Logs folder. This will take a bit to create, so be patient.

StorageGroup-2

Rename database edb file
Rename Mailbox Database.edb to Mailbox Database.edb.oldCreate a new Database in the storage group you just created called Mailbox Database.edb, DO NOT mount the database.
Rename the Mailbox Database.edb.old back to original, remove .old

Go into the database properties and adjust the quota’s and make sure you check ‘This database can be overwritten by a restore’

DBProperties

Mount the Database

Run the following powershell command to re-home the mailboxes from the old mailbox server to the new mailbox server

[PS] C:\Windows\system32>Get-Mailbox -Database “MBX1\Storage Group
\Mailbox Database” | where {$_.ObjectClass -NotMatch ‘(SystemAttendantMailbox|ExO
leDbSystemMailbox)’}| Move-Mailbox -ConfigurationOnly -TargetDatabase “EXMBX4\Storage Group\Mailbox Database”

Run the following command to show all the mailboxes in the database and export to a txt file so you know who’s mailbox is back online

[PS] C:\Windows\system32>Get-Mailbox -Database “Storage Group\Mailbox Data
base” > C:\DB1.txt

Outlook Connection Status

What are good numbers to have in  Outlook Connection Status?
These numbers are for cached mode.  Online mode you will want 50ms or lower otherwise Outlook will constantly be hanging to grab the information from the server.

  • Max Avg Proc Time (Exchange RPC Latency) = 25ms
  • Max Avg Resp Time (Exchange RPC Latency + Network Latency) = 325ms

OutlookConnectionStatus

To get to the Outlook Connection Status:

  1. Launch Outlook
  2. Hold Ctrl and left click on the Outlook system tray icon
  3. Select Connection Status…