Drives getting filled up soon because of the Usage Data–THE TIMER

I recently faced this problem in my production farm where the D: drive which we usually store log files from various technologies running on SharePoint server, gets filled up too soon. We have a directory called D:\LogFiles which has various subfolders for storing log files that get created from SharePoint, IIS, log4net (custom logs) and also the Usage data generated by health and usage feature of SharePoint. The drive was almost 50GB in size and at any point of time the complete LogFiles folder generally will be of size 20 GB in total. The majority of those logs will be ULS because we have enabled verbose logging.

When we got a disk space low alert from our monitoring solution for D: drive, we initially thought that the culprit was IIS logs since the SharePoint logs and the log4net logs are configured for rotation. We cleaned up majority of those logs and even cleaned up old, unused setup files etc from D: drive. However, the problem started to repeat again and there is nothing else to delete from that drive.

We did a deep dive to find out that D:\LogFiles\UsageData contains large number of files and they were never getting rotated. We know we have enabled health and usage data collection and it should collect the data but it should not store the data on the drives. Also other servers in the farm (total of 7 servers in the farm) were not affected. That had given us a clue that there was something wrong with THE TIMER service.

We know the timer is the most important service of the SharePoint architecture. It makes sure all tasks with in SharePoint farm are carried out at the designated times. I really think of it as the heart of the SharePoint body since it mobilizes the rest of the SharePoint tasks and activities in the farm of timer jobs. If the timer is not working, multiple things start happening on the server and the farm like the problem described above. There is a timer job called Microsoft SharePoint Foundation Usage Data Import which will actually import all the usage data into the database and deletes those temporary files from the server. I checked the timer services from the Services console but it was running. However its not doing its job properly. So we restarted the timer service, which cleaned up the usage logs like eating up a piece of cake. Once again, the importance of timer service is understood.

Hope this helps in your troubleshooting and thanks for reading!!

Deleting customised web applications in SharePoint 2010 – HTTP 404 Error

General Introduction

Deleting SharePoint 2010 applications is always an easy task. At least for the web applications with site collections having no major customisations and features activated on them, it is moderately easy to go to Central administration -> Manage Web Applications -> select the required Web application and click delete button from the ribbon. It immediately prompts to select the 2 major components to be deleted which are the content database and the IIS container sites. You can select either one of them or both and click OK to delete the required web applications. Alternatively, Windows PowerShell can be leveraged to remove the web application by using the Remove-SPWebapplication cmdlet with -DeleteIISSite and/or –RemoveContentDatabases option(s) as required.

Problem Description

However, if your farm is heavily customised with many farm level solutions used by various web applications, it is not easy to delete a web application. This is what I have learnt from my experience. When I tried to a web application which is heavily customised and is also extended to a different zone with a different type of authentication, the application is not getting deleted although I tried to delete the web application in various ways mentioned above. The error I was getting is a typical SharePoint error page with a correlation ID and a message “404: File not found”. At the first second, I thought “What! A 404 error?” Then I checked the IIS container sites which are removed from the server. But the web application continued to appear in the web application management page in the central administration.

Troubleshooting & Resolution

I tried to troubleshoot further to check if there was timer job still running or hung by any chance. I browsed to Timer job definitions in the Monitoring section of central administration. However, this page started to show the same error. It was at this point that I realised there was something else going on. Time for ULS logs!! I checked the correlation ID in the ULS log and found that there were few custom application DLLs missing in the registry. These DLLs were part of the solutions (wsps) that were deployed to the web application to be deleted. I had no idea why the DLLs got knocked out of registry since I was not removing the WSPs. The WSPs which were deployed only to the web application, that I was removing, changed their status to “Not Deployed” state but the DLLs were never knocked out. I could not find a clue for why they got knocked out. I had manually placed those missing DLLs listed in ULS logs back in the registry in all the servers of the farm and did an IISReset. Then the timer job definitions page started working. And, we were able to successfully delete the web application too. The manage web applications page did not display the web application this time.

Looking back into what has happened, when we have a farm with highly customised applications, I think it is always recommended to deploy the DLLs (WSPs) to the bin directory of the application’s virtual directory rather than the system’s assembly.

I hope this post helps in resolving this type of issues that you might be facing.

HTTP 500 Internal Server Error–SharePoint 2010 Applications

Among the various HTTP return status codes that we receive, the two most frequent ones are 401s and 500s. While the 401s are more related to authentication and authorization, the 500s are because of problems within IIS. I am listing various things to check and ways to troubleshoot the HTTP 500 errors for IIS web applications in general and SharePoint 2010 applications in particular.

• The first thing you want to check when you receive this error, when you hit the root site of the web application, is the status of the application pool of the website. Is the application pool running?Crashed? Is the application pool properly associated with the web application?
• There is could be a possible 500 server errors in some scenarios based on the load balancer configuration. There are few settings for IIS application pools that that recycle the application pool at preconfigured intervals, by default every day. When the web applications are hit after recycle or the pool dies out the memory, I have learnt that the IIS 7.5 takes some time to WARM UP and if the load balancer’s time out is not configured to be greater than the warm up time, it is going to tell back to the client that there is some error. So make sure you have properly configured those time-outs and if you see such type of issues, just set up a warm job that keeps on pinging the URL of the web application.
• You can see the trace logs in SharePoint 2010, whether there are any assemblies that got knocked out of the assembly. If so, make sure they are added back and reset IIS to get the replaced .dll picked up.
• If you do not see any of these work, Microsoft has given a nice feature with in IIS to trace out the failed requests. Probably this should be used by every single administrator who works on IIS. This is called “Failed Request Tracing Rules” available under IIS section in the Features view of each web site.

1. Enable the Failed Request Tracing logs and configure the location to store the trace logs.
2. After that, double click on the feature to add the trace rules. Click on Add button to add the content source, select All Content. Click Next.
3. Select the status code option and give 500 as the status code. Click Next.
4. Select all the provider options shown in the screen and click finish. Now a trace rule gets added in the list of the feature.
5. Go ahead and hit the application in the browser and see the 500 error again.  Couple of files get created in the directory where you have configured the logs should go.
6. Open the xml file in Internet Explorer (MUST). If you open the xml in any other xml editor, you may not see the XML styles getting applied.

The opening page will show the errors and warnings that IIS has encountered while processing the HTTP request. Those traces will provide you a direction to troubleshoot the issue.

In particular to SharePoint 2010, the issue I faced was that the My Sites web application was not coming up and it was giving the 500 internal server error. When I did the Failed Request tracing explained above, I have seen couple of warnings in the log as shown below.

I opened the applicationHost.Config which is the one of the IIS metabase files where the details about Sharepoint14Module were present.

<add name="SharePoint14Module" image="C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\isapi\owssvr.dll" preCondition="appPoolName=S………

The preCondition property value of this key is missing my application pool name in that list. I have added the application pool of the my sites web application and it started working.

Hope this helps for your troubleshooting and thanks for reading.

“A web.config modification operation is already running” error in SharePoint 2010

It is pretty common that most of the enterprises do customize the SharePoint platform to a great extent by deploying solutions (WSPs) to the farm and associating features to those WSPs. When a feature is getting de-activated or activated, a web.config file change is triggered in the from of a timer job. This happens for almost all such operations. The file modification is done by an internally created timer job. It shows up as ‘job-webconfig-modification” when you execute the following command in SharePoint snap in loaded PowerShell console.

Get-SPTimerJob –Webapplication <ApplicationName>

There is another place where you can see the same timer job in “Run once” state. It is the Central Administration –> Monitoring –> Timer Jobs section. The job name should be visible as “Microsoft SharePoint Foundation web.config modification” job. This job should come up in the running job list for a while and then disappear as it gets removed after the modifications are done. As mentioned before, this is a run once job.

Getting into the problems, sometimes while activating or deactivating the features, you might receive this error (which is understandable from ULS logs) that a web.config modification is already running and this operation cannot be completed. This can be because you are trying to deactivate or activate features so quickly that you are not giving SharePoint the necessary time to complete its previous action. In some cases, although your previous action is complete a long time back, this still gives the same error. When I faced this error, I had no clue about what is happening because there is no feature activation deactivation that I did earlier nor I see the timer job in the PowerShell console. However I have learnt that the “Microsoft SharePoint Foundation web.config modification” job in Central Administration is present the job definition list and its schedule is stated as “Disabled”. I was not sure why it was not deleted after the feature deactivation/activation activity that I did before. It just remained there in the list disabled and it was stopping me from doing the current operation of activating features. I cannot take any action on that job except for leaving it like that DELETE it. I deleted the job and my current operation of feature activation and deactivation is now working :-).

So in case you are having similar issue and find such scrap job in your scheduled timer jobs list, please go ahead and delete it if it is in disabled state. This should work. I hope it helps and thanks for reading.

SharePoint 2010 ULS (Trace) Logs Not Working

This is one of the most common problems, I believe, with SharePoint 2010. You set up your brand new farm and configure the required logging levels in Central Admin – Configure Diagnostic logging page. However, you will notice that the log files get created in the folder you assigned the logs to be written to with no data inside them. The logs get rotated as usual but you will find the log file size as 0KB. The files are empty. Here is a solution for such problem scenarios.

There are absolutely 2 (or 3 more exactly) places where SharePoint administrators are exposed to tweak Logging mechanism of SharePoint out of the box. The Central Admin Monitoring Tab has a link to configure diagnostic logging settings which is the first place of interest for any administrator to change the logging settings. The other place is the “SharePoint 2010 Tracing” Windows Service which the actual engine that controls the logging on each individual server of the farm. This is the first point of check for the admins to see whether all the settings in Central Administration are configured properly and if this windows service is running. The same can be checked through power shell commands.

If you still see the same error, then there is a problem with security. In my experience, out of 5 farms that I have installed, these security settings were not properly implemented by SharePoint 2010 installation/configuration in one farm. I don’t know why. However, in order to fix it, please get the credentials with which the service SharePoint 2010 Tracing service is running. Make sure the user account is added to the local group “Performance Log Users”. It should be added to “Performance Monitor Users” local group as well. Restart the windows service after this change and that’s it. This should fix the security issues with the logs and logs should start rolling as per your configuration in central administration.

I read somewhere in the internet that even the application pool accounts should also be added to these security groups but I do not think so and I never had to do it to get the logs rolling.

Thanks for reading and hope this information helps in resolving any SP2010 logging issues you are facing.

Solution Deployer for Sharepoint 2010

DOWNLOAD HERE

How it works

·         After you install, Solution Deployer for SharePoint 2010 provides a config.xml file in which there are various values to be populated as per your environment. This is a onetime task and you can populate multiple environments too. More details about this xml and how to fill it following sections.

·         Copy the WSP files to a location on any of the servers or all the servers along with the required text files.

·         Run the tool by selecting the required options in the GUI. Check the status of the deployment in the status update pane.

That’s it!!

Prerequisites

•       SharePoint farm should be reachable from the computer the tool is running.

•       PowerShell Remoting should be enabled on both the SharePoint servers and the computer on which the tool is running.

•       The tool is supported to run on Windows 7, Windows Server 2008 and Windows Server 2008 R2. (CredSSP authentication support)

•       Make sure the text files for each set of WSPs/change is available as described in user guide section.

Installation (One Time)

Installation is pretty much easy. Download the zip file to any server or computer from which you can reach the farm by executing the tool. Here is the step by step process in detail.

•       Extract the contents of the zip file and leave the directory structure intact.

•       Open PowerShell console with administrative privileges and execute the script ‘SPDeployerPreRequisiteSetupClient-OneTimeRun.ps1’ under “FirstTimeSetup-OneTimeExecution” folder. Remember that this should be run on the computers on which this tool is running.

Type Yes or Run Once where ever the script prompts you to do so.

•       Copy the file “SPDeployer-PreRequisiteSetupServer-OneTimeRun.ps1” to all the application and web front end servers of the farm to a directory.

•       Log on to all the application and web front end servers of the SharePoint farm and open

PowerShell console with administrative privileges. Change the directory to the place where

you copied the script to “SPDeployer-PreRequisiteSetupServer-OneTimeRun.ps1” and execute the same. Type Yes or Run Once where ever the script prompts you to do so.

That’s it. The environment is ready to run the tool.

Configuration – Modifying Config.xml (One Time, and later, Need based)

This section basically describes about setting up config.xml. This xml contains all the SP2010 farm related information required for successful functioning of the tool. This is an Easy to fill xml. Caution should be taken not to change any tags. Only the values should be changed according to your needs. The points below explain about various tags. The tags missing in these point are obvious and need not be explained.

The XML is structured this way.

Parent Node (SHAREPOINT_CONFIGURATION) -> Environment Classification ->Application

Classification in each environment

1)       Fill the “EnvName” value as your environment name. For example, Development, Disaster Recovery etc.

2)       Fill the value between tags as the enterprise domain where the sharepoint farm resides and does user authentication.

3)       Fill the value between tags as the username with farm admin privileges or privilege to deploy WSPs and manage features. Do not mention

‘domain\username’ format but just ‘username’

4)       Fill the value between tags as the encrypted password of the farm admin user mentioned above. The password can be encrypted using

“encryptString.ps1” PowerShell script present in “EncryptionTool” directory of the extracted and installed folder. Right click on encryptString.ps1 and select “Run with PowerShell”. Provide the plain text password at the prompt and hit Enter. You will get the encrypted string. Copy and paste the encrypted string in between these tags in the xml.

5)       Fill the value between tags with all the application and web front end server names in comma separated format. The first server mentioned in the csv server list is the server used by the tool to do all the deployment work.

6)       Fill the value between tags with UNC or absolute path where you want to store all the logs of this tool. This logs who is working on the deployment and all the text shown in the status update pane.

7)       Fill the value between tags with the central administration solution management page URL.

8)       Fill the value of “AppName” in tags with the web application name.

9)       Fill the value of tags with the absolute path on the server where you store all the WSPs and the related text files. More details about the text files in the User guide.

10)   Fill the value of with the URL of the web application as you see in central administration.

11)   Fill the value of tags with the relative path of the IIS virtual directory of the web application.

The tags can be of any number based on your setup and so the applications under tags. 

Quick User Guide

The following are the quick steps to jump start using the tool.

1)      Maintain a common folder (deployment scripts base path) on any server of sharepoint farm and copy the WSPs deployed to that folder

2)      Create and copy 2 text files into the same directory whose names are of format “ChangeNumber_WSPList.txt” and “ChangeNumber_Features.txt” where change number is the actual change number for this release.

3)      List the wsps to be deployed in ChangeNumber_WSPList.txt comma separated with URL to which the WSP should be scoped. If it is global do not mention anything.

4)      List the feature IDs associated with the WSPs to be activated and deactivated in the ChangeNumber_Features.txt file.

5)      Run the SharepointDeployerV3.1.exe and fill in the required details. Click Apply Config button.

6)      This populates the WSP text files to be used for the deployment and features for activation and deactivation. Select the right options for the deployment and click Deploy.

7)      Check the status pane for updates.

User Guide

In order to maintain consistency, especially when deploying same set of WSPs across various environments, the following process works well but you can modify it as you need. The following steps are explained considering a general scenario of every enterprise into consideration. It might differ a little from what you do but it can always be customised.

Consider there is a requirement from business to sharepoint developers and the developers have written code in the form of 3 WSPs which creates 2 features to a sitecollection of web application webapp1. However all the WSPs are existing WSPs in production and they have to be replaced with these new ones. They want to deploy these 3 WSPs in Dev environment first. The enterprise is using Solution Deployer for Sharepoint to do the WSP deployments. Considering the enterprise practises ITIL Change and Release management, this is what the developer does.

1)       The developer creates a Change record and the change number is RFC12345. This is used to track the deployment till it reaches production from development.

2)       While building WSPs, the developer also creates two text files with following name format. This is a must.

a.        RFC12345_WSPList.txt

b.       RFC12345_Features.txt

The WSPList text file is used while retraction and deployment and Features.txt file is used during feature activation and deactivation. You can skip creating any of these files if you are not doing retraction/deploying or activation/deactivation.

3)       The content of RFC12345_WSPList.txt is as follows.

Basically it will contain the list of WSPs along with any specific URL to which the WSP need to be deployed to. The URL and the WSP are separated by a comma. Also if the WSP has to be deployed globally, there is no need to place a comma.

4)       The content of RFC12345_Features.txt is as follows.

 

Basically, this will contain the Feature IDs or Feature Names of the features that are going to deactivated and activated in this particular deployment and the scope of the feature. All the features listed will be deactivated and activated if the option is selected in the tool. If no scope is mentioned, it gets deactivated the default scope.

5)       Now, the directory in which you copied the WSPs and the above text files is the deployment scripts base path and the absolute path of this directory should be placed in the tags in config.xml file.

6)       Launch the tool by double clicking on SharepointSolutionDeployerV3.2.exe file.

7)       Provide the RFC Number in the first field. For example RFC12345

8)       Select the environment

9)       Select the application of the environment to which deployment is happening

10)   Click on Apply Config to populate all the fields in section below. The fields will be auto populated based on the process that we have followed earlier. In cast the file names or paths are different, you have an option to change them.

11)   Check the boxes for the actions that you want to carry out and click on Deploy button.

12)   The tool will complete all the required validations during execution and complete the required actions that are selected. The status is shown in the status field.

Points to Note:

a)       The tool is basically a powershell script wrapped up in an exe. So there will be a console window that opens along with the tool. The console window displays any errors during execution and also displays overall status messages as well.

If you do not want the console to open and you want just the GUI, execute the exe with – noconsole option.

b)       Make sure you populate the encrypted password and other values in config file properly. Any mistakes in config file will lead to mistakes and errors. The encrypted password feature is to enhance the security.

c)       If you do not want to follow the process mentioned above, you could still use the tool by modifying the text file paths on the server during execution.

d)       In case you want to use different text files for retraction and deployment, you have an option in the tool to retract using a different file. By selecting the option, the tool will consider the file that you mention here for retraction and the actual WSPList file for deploying WSP files.

e)       There is an option for config file changes which halts the execution of the tool in case you want to make any other changes after deactivation/retraction/deployment/activation steps. This might include changing any web.config entries or owstimer.exe.config entries etc. You can select the options on prompt to continue further with IIS reset and timer restart.

f)        The sequence of actions carried out by the tool is 

a.        Deactivate features

b.       Retract Solutions

c.        Deploy solutions

d.       Activate features

e.       Config File Changes

f.         IIS reset

g.        Timer restart

If you do not select any option in the above list, the tool will skip that action but the sequence is maintained.