During our last Chalk Talk session @ Techdays belgium someone asked a question about Exchange Server Performance Troubleshooting.
The problem was that Clients were experiencing the famous “Requesting Data” notification in Outlook.
They had different Outlook versions throughout the company, Outlook 2002 aka XP, 2003 & 2007.
The Outlook XP users were experiencing the most problems.
The question was how can we troubleshoot this?
Looks like a simple question right? There are so many things that can cause performance issues on an Exchange server.
Memory, CPU, Storage, 3rd Party Server Applications, 3rd Party Client Applications, Mailbox size, Mailbox items, Server views, Network, Active Directory …
You see there is no single answer to this question…
Any of the following sections can help you in the design/validation/troubleshooting process
1. Outlook and Exchange Server Compatibility
Let’s look at a roster of Outlook clients supported for Exchange 2007
So Outlook 2002/XP is supported for Exchange 2007.
2. Exchange Server Sizing
Processor and memory configurations for Exchange Server 2007 Server Role
3. Storage Related Information
Planning/Optimizing Storage Configurations
Exchange 2007 Mailbox Server Role Storage Requirements Calculator
- How to Align Exchange I/O with Storage Track Boundaries
With a physical disk that maintains 64 sectors per track, Microsoft Windows always creates the partition starting at the sixty-forth sector, therefore misaligning it with the underlying physical disk. To be certain of disk alignment, use Diskpart.exe, a disk partition tool. Diskpart.exe is a tool provided by Microsoft in the Windows Server 2003 Service Pack 1 support tools that can explicitly set the starting offset in the master boot record (MBR). Setting the starting offset correctly will align Exchange I/O with storage track boundaries and improve disk performance. Microsoft Exchange Server 2007 writes data in multiples of 8-kilobyte (KB) I/O operations, and I/O operation to a database can be from 8 KB to 1 megabyte (MB). Therefore, make sure that the starting offset is a multiple of 8 KB. Failure to do so may cause a single I/O operation spanning two tracks, causing performance degradation.
- Storage Technology
4. Pre-Deployment Testing/Verification tools
- Microsoft Exchange Server Profile Analyzer
Collect estimated statistical information from a single mailbox store or across an Exchange Server organization.
- Exchange Load Generator
Exchange Load Generator is a simulation tool to measure the impact of MAPI, OWA, IMAP, POP and SMTP clients on Exchange servers.
- Microsoft Exchange Server Jetstress Tool
Simulate disk I/O load on a test server running Exchange to verify the performance and stability of your disk subsystem before putting your server into a production environment.
5. Active Directory Testing
- Active Directory Performance Testing Tool (ADTest.exe)
ADTest.exe is an Active Directory load-generation tool that simulates client transactions on a host server to assess the performance of the Microsoft® Active Directory™ within Microsoft® Windows® Server 2003 and Microsoft® Active Directory Application Mode™.
6. Exchange Analysers
Read about the features, abilities, and future vision for the Exchange Analyzer family. Visit the Exchange Team Blog for the details.
- Exchange Remote Connectivity Analyzer
The Exchange Remote Connectivity Analyzer tool is a Web-based tool designed to help IT Administrators troubleshoot connectivity issues with their Exchange Server deployments. The tool simulates several client logon and mail flow scenarios. When a test fails, many of the errors have troubleshooting tips to assist the IT Administrator in correcting the problem.
- Exchange Best Practices Analyzer
The Microsoft Exchange Best Practices Analyzer is designed for administrators who want to determine the overall health of their Exchange servers and topology. The tool scans Exchange servers and identifies items that do not conform to Microsoft best practices.
- Exchange Troubleshooting Assistant
The Exchange Troubleshooting Assistant programmatically executes a set of troubleshooting steps to identify the root cause of performance, mail flow, and database mounting issues. The tool automatically determines what set of data is required to troubleshoot the identified symptoms and collects configuration data, performance counters, event logs and live tracing information from an Exchange server and other appropriate sources. The tool analyzes each subsystem to determine individual bottlenecks and component failures, then aggregates the information to provide root cause analysis.
7. Performance Troubleshooting tools
- Microsoft Exchange Server User Monitor
Use the Microsoft Exchange Server User Monitor to gather real-time data to better understand current client usage patterns, and to plan for future work. Administrators can view several items, including IP addresses used by clients, versions and modes of Microsoft Office Outlook, and resources such as CPU usage, server-side processor latency, and total latency for network and processing with Outlook 2003 version MAPI.
- Description of the Exchange Server Performance Troubleshooting Analyzer tool
Run the Exchange Server Performance Troubleshooting Analyzer tool when the problem is occurring. The tool will complete the following tasks:
- Identify performance problems that may cause Microsoft Outlook to display the RPC Cancel Request dialog box.
- Analyze the data in the Microsoft Exchange Server User Monitor tool (Exmon.exe) event trace log file (.etl) to determine whether a load issue is caused by one or two users or by many users.
- Performance Analysis of Logs (PAL) Tool
The PAL (Performance Analysis of Logs) tool is a new and powerful tool that reads in a performance monitor counter log (any known format) and analyzes it using complex, but known thresholds (provided). The tool generates an HTML based report which graphically charts important performance counters and throws alerts when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams and members of Microsoft support, but continue to be expanded by this ongoing project. This tool is not a replacement of traditional performance analysis, but it automates the analysis of performance counter logs enough to save you time. This is a VBScript and requires Microsoft LogParser (free download).
- ExInsight for Microsoft Exchange
ExInsight is a free monitoring and statistics tool that provides a real-time view of the internals of Exchange Server transactions and allows you to evaluate individual user’s usage of Exchange Server.
This tool is especially useful for Exchange administrators and developers. By using this tool, you can gather real-time data about the users in an Exchange environment and learn more about the inner workings of MAPI, OWA, POP3, IMAP4, NNTP and other protocols. ExInsight helps you to understand how individual users affect the performance of an Exchange Server, analyze application usage of Microsoft Exchange, and guide troubleshooting of applications that depend on Microsoft Exchange.
8. Performance Issues
This section provides information about how to isolate performance degradations in Microsoft Exchange Server 2007. Information is also provided about tools that may assist you in performance troubleshooting.
- Understanding the Performance Impact of High Item Counts and Restricted Views
This topic will help you understand, diagnose, and resolve Microsoft Exchange Server 2007 performance issues related to high item counts in critical path folders and restricted view requests when you use Microsoft Office Outlook running in online mode, Outlook in cached mode with delegate access, and Outlook Web Access. The critical path folders are the Calendar, Contacts, Inbox, and Sent Items folders. Restricted views are data views that restrict information based on search criteria that result in views of only a subset of items in a folder. Performance issues related to these situations are frequently related and can become visible to end-users in the form of slow client access. It only takes a few users who have abnormally high item counts in their critical path folders to cause performance issues which are felt throughout your whole Exchange organization.
- Troubleshooting Slow RPC Request Processing Issues
When you use Microsoft Office Outlook in MAPI mode, Outlook executes client operations as remote procedure calls (RPCs) between the client and the server. If the user is running in online mode, these RPC calls occur synchronously. Any delay by the server in fulfilling these synchronous requests directly affects the user experience and the responsiveness of Outlook. In contrast, most operations that are performed when you run in cached mode occur against the user’s local copy of the mailbox or are issued to the server in the form of asynchronous (background) RPCs. Generally, asynchronous RPCs do not affect the responsiveness or overall experience of the Outlook client itself.
- Outlook users experience poor performance when they work with a folder that contains many items on a server that is running Exchange Server
- Microsoft Exchange Server 2007 Management Pack for System Center Operations Manager 2007
This management pack includes scripts and rules to effectively monitor Exchange 2007 and report on performance, availability, and reliability of its server roles.
10. Some very interesting blog reading:
- ‘Requesting data from the Exchange server…’ or ‘Outlook is trying to retrieve data…’ client messages
- Recommended Mailbox Sizes