FileSystemWatcher

(Originally published: 10 June 2007)

The FileSystemWatcher (FSW) component available in .NET is being used more and more every day. Unfortunately FSW does not work quite the way you would expect. This article will discuss how to use the FSW in a .NET application. An additional problem with using FSW is performing complex tasks without impacting the performance of the system overall. This article will discuss a basic approach to working with FSW while maintaining good performance.

What Is the FileSystemWatcher?

Before discussing how to use the FSW it is important to understand what it is and what it is used for. The FSW is a component used to monitor the file system for file or directory changes. The FSW can be used to receive notifications when files or directories are created, moved, deleted or updated. Although it is generally only useful in Windows applications it can be used in any type of application including web applications.

Under the hood the FSW relies on the underlying file system in Windows. As a result the events raised by FSW are low level. However they are accurate and fast, being raised by the file system as soon as the operation occurs. No buffering of events occurs either. This will prove to be problematic as we will see later. FSW identifies the following events for both files and directories:

Create
Delete
Change
Rename

To avoid raising too many events the FSW also supports filtering of files and directories based on a file path or a file mask. This is very useful as the file system is used a lot. It would be wasteful to receive every event raised by the file system if all we cared about was the modification of a specific file. To demonstrate how to use FSW refer to the simple file monitoring application. FileMonitor will monitor any folder it is pointed at. As events are raised it will log the event data to a simple UI control. To keep things simple the application simply exposes a PropertyGrid for interacting with the FSW properties. Run the sample application as you read through the article so that you can better understand what is going on.

Configuring the FileSystemWatcher

The FSW is a standard .NET component. It can be dragged from the Toolbox in Visual Studio and dropped onto a form or created programmatically in code. It is generally easier to create and use the FSW programmatically rather than dragging and dropping it onto a parent form. For the sample application however dragging and dropping the component from the Toolbox works just fine. The FSW resides in the System.IO namespace since it is a general component rather than a WinForms component.

Before the FSW can be used you must configure the events you are interested in and apply any filtering desired. Once the FSW is configured you must enable it using the EnableRaisingEvents property. This property starts (or stops) the FSW. We do not want to set the property until we have fully configured the component otherwise we might miss some events.

The Path property indicates the folder to be monitored. If the IncludeSubdirectories property is set then any subfolder is also monitored. Use this property with care. If you set Path to a folder with many subdirectories you might get more events then you want.

Set Path to some folder on your machine (for discussion purposes assume c:temp). Leave IncludeSubdirectories as false and set EnableRaisingEvents to true. The FSW is now watching the given folder for any changes made to the file name, directory name or last write time. Go to Explorer and copy a file into the folder. Notice that several events were generated. We will discuss why shortly. Rename the file and then delete it. In each case more events are raised. As you can see from the list the full path to the file is given each time along with the event that occurred.

Now set IncludeSubdirectoriesto true and create a new folder in the folder from earlier. Copy a file from the original folder to the new folder. In each case you should see some events being raised. It appears that you can change the properties of the FSW without

FileSystemWatcher fsw = new FileSystemWatcher();
fsw.Path = @”c:temp”;
fsw.IncludeSubdirectories = true;
fsw.SynchronizingObject = mainForm;
fsw.EnableRaisingEvents = true;

Listening For Events

Since the FSW is a standard .NET component it raises events in the same way as everybody else. The FSW only defines five events of interest.

We will ignore the Error event. Each event does exactly what you think it does. All the events except for Renamed have an FileSystemEventArgs argument. This argument specifies the following useful properties.

ChangeType – An enumeration identifying the change (in lieu of the event).
FullPath – The full path and name of the file or folder causing the event.
Name – The name (without path) of the file or folder causing the event.

The Renamed event has an argument of type RenamedEventArgs which derives from FileSystemEventArgs. It adds or modifies the following properties.

FullPath – The full path to the new file or folder.
Name – The name of the new file or folder.
OldFullPath – The original path of the file or folder.
OldName – The original name of the file or folder.

In the sample application an event handler is defined for each event. A formatted message is generated in the list box containing the event arguments.

I know what you are thinking. “What a minute! You have a threading issue in your code”. Since the FSW is not a WinForms component and is called by the operating system there is a good chance that the event will be raised on a non-UI thread. You are absolutely correct…sort of. The event will indeed be invoked from a non-UI thread. However since the component is generally designed for WinForms applications it is built to work without too much effort. Internally the component will transition to the UI thread before it raises the event. Therefore you can safely access the UI from within any event handler hooked up to the FSW.

FileSystemWatcher fsw = new FileSystemWatcher();

fsw.Changed += OnFileChanged;
fsw.Created += OnFileCreated;
fsw.Deleted += OnFileDeleted;
fsw.Renamed += OnFileRenamed;
fsw.EnableRaisingEvents = true;

Filtering Events

The FSW supports two different types of filtering: file/folder name and change type. The Filter property is used to filter the events based upon a file/folder name. It supports standard wildcard characters so if you want to monitor a specific file specify the file name in the Filter property. If you want to monitor a type of file (such as text files) then enter the appropriate mask (such as *.txt). Note that only a single mask can be used. Do not attempt to use a semicolon to separate multiple file masks because it will not work.

The other filtering option, change type, is available strictly for Changed events. The Changed event is raised when anything about the file changes including its name, size or even its attributes. Generally all you care about is if a file is modified. Receiving a change notification when a file changes from readonly to read-write is wasteful. Enter the NotifyFilter property. This property is a list of one or more enumerated values indicating the types of information for which a change notification should be generated. The default is whenever the name or last write date changes. You can watch for other changes as well such as attribute changes or even when the file was last modified.

The only downside to the NotifyFilter is that when the Changed event is raised there is no way to tell why it was raised. Ideally the ChangeType property on the event argument would indicate the exact change that occurred but it does not. It will always say the same thing. Therefore if you care about the exact change that was made you will need to keep track of that information yourself.

For the Renamed event if either the old or new file matches Filter then the event is raised.

Demystifying FSW Events

One of the most confusing issues with using FSW is figuring out why certain events are not raised. It is a common misconception that if you modify a file that you will receive a change event. This is not necessarily true. We will now discuss a few common file monitoring scenarios and discuss how to handle each one.

Monitoring for Creation

There are basically two ways to create a file: create a new file or rename an existing file. To successfully detect the creation of a file you should monitor the following events.

Created
Renamed

Created is raised in all cases if you have no filter and only if FullPath matches the filter otherwise. For Renamed you have to do a little more work if you are using a filter. If FullPath matches the filter then a file has been renamed to a file type that you care about. If OldFullPath matches the filter then a file has potentially been renamed from a file type that you care about. Notice I said potentially. You can rename a file without changing its file type. Therefore use FullPath first and only use OldFullPath if needed to determine what the event is signifying.

private void OnFileCreated ( object sender, FileSystemEventArgs e )
{
Display(“Created file {0}”, e.FullPath);
}

private void OnFileRenamed ( object sender, RenamedEventArgs e )
{
Display(“Renamed file {0} to {1}”, e.OldFullPath, e.FullPath);
}

Monitoring for Deletion

As with creation there are two ways to delete a file: delete the file or rename an existing file. To successfully detect the deletion of a file you should monitor the following events.

Deleted
Renamed

Deleted is raised in all cases if you have no filter and only if FullPath matches the filter otherwise. Renamed works the same way as it does for file creation.

A special case exists for deletion of a directory with subdirectories and/or files. A deletion event will be raised for each file in the directory (and each subdirectory if IncludeSubdirectories is set). Additionally if subdirectories are being monitored then a change event will be raised for each directory in which a file or directory was deleted.

private void OnFileDeleted ( object sender, FileSystemEventArgs e )
{
Display(“Deleted file {0}”, e.FullPath);
}

Monitoring for Changes

Think about how a program might save data to an existing file called mydata.dat. The naïve approach would be to open the original file and overwrite whatever was there. Although this is fast and easy to do if something goes wrong during the save process not only are the changes lost but so is the original file. Almost everyone has experienced this at least once. Now think about the events that would be involved.

Change for mydata.dat
Change for mydata.dat
…

The “superior” approach would be to create a temporary file, save the file to the temporary file, delete the original file and then rename the temporary file to the original file. Although there is more work involved and an opportunity for failure at no point would the original and changed data be lost. You could expect to see the following events.

Create of someTemp.tmp
Change of someTemp.tmp
Change of someTemp.tmp
…
Delete of mydata.dat
Rename from someTemp.tmp to mydata.dat

If you only monitor events for mydata.dat you would completely miss the saving of the file. Therefore you will often need to use heuristics to decide what events to process and what events to ignore. The following events should be monitored.

Changed
Deleted
Renamed

Changed is raised whenever the file changes. Remember that the Win32 API is used behind the scenes and it is low level. Therefore you can receive several Changed events for a single logical change in an application. You must handle this case.

Deleted is raised when the file is deleted. This is normally considered to be a change of the file although how it is handled is application specific.

Renamed is the most difficult event to handle. Go back to the file change discussion of earlier as you read this description. If FullPath is the file of interest (mydata.dat) then you can assume that the superior approach was used. Treat this case as though the entire file has changed. If OldFullPath is the file (mydata.dat) then treat it as though a deletion has occurred.

private void OnFileChanged ( object sender, FileSystemEventArgs e )
{
Display(“Changed file {0}”, e.FullPath);
}

Locked Files

You will not get to far into using the FSW before you will want to do something with the file that you are receiving the event about. You might want to copy the file to another directory or perhaps read the new file contents. Remember, again, that the events are ultimately raised in the low level details of the operating system. There is a real good chance that the application that is manipulating the file still has a lock on it. Some applications allow shared read access when they update the file while other applications deny all access to the file. Therefore it is generally a good idea to assume that, while processing the event, the file in question is locked. This puts quite a limitation on what you can do with the file.

To work around a locked file there is little option but to wait until the lock is released. Rarely is the lifetime of a lock predictable so generally the file has to be polled. Depending on the application polling every second or every couple of seconds should be sufficient. Polling generally involves just trying to get access to the file. Since file operations generally throw exceptions when things go wrong exception handling will probably be needed. It will not perform well but we will handle that later.

A final issue with dealing with locked files is the problem of leaked locks. An application may refuse to release a lock for whatever reason. It is generally not a good idea to continually poll for a file. After some point it is best to just give up. Therefore it is a good idea to define a retry count for each event. After the retry count is exceeded report an error and ignore the event. Depending on the application this could require that all additional events for the same file be ignored as well.

A complication added by locked files is how to handle other events. Ideally a single locked file should not stall all file processing. If file A is locked events for file B should still be processed. This will quickly become complicated due to file reuse. File reuse occurs when a file name is reused for several different files. Examples of file reuse include renaming a file and deleting a file and then recreating it (i.e. changing a file). Here is a sample of events that can cause file reuse.

File B is created
File B is changed
File A is deleted
File B is renamed to file A
File B is created
File B is changed

If file A is locked then file B can not be renamed nor can any subsequent events against file B be handled until file A is complete. Tracking this can become quite complex. In most applications it is not worth the effort therefore processing events in order (with retry counts) is a simple, reasonable requirement.

Buffering of Events

The sample application simply logs messages to the UI. In general however you will be doing far more work. Work that will take a reasonable amount of time (such as retry counting). The FSW communicates to the underlying file system through a shared buffer. This buffer is limited in size. If more events come from the file system than the FSW can handle the buffer will overflow. In this case two things will happen: you will miss events and the Error event will be raised.

If you start missing events then there is little you can do about it. Therefore the best option is to ensure that any FSW event handler processes the event as quickly as possible. This pretty much mandates that you push the actual event processing to a secondary thread. Moving work to a secondary thread is pretty standard fare in most applications. Therefore we will forego the usual discussions of thread safety and synchronization and instead focus on solving the general problem of handling FSW events on a secondary thread.

The general algorithm would work as follows.

private void OnSomeFswEvent ( object sender, FileSystemEventArgs e )
{
    //Create instance of event data structure containing the event type and file(s) of interest
    //Lock the shared queue for writing
    //Push the event data onto the queue
    //Unlock the shared queue
}

private void SomeWorkerThread ( )
{
    //While not terminating sleep for a fixed period of time
    //    While there are events in the queue
    //        Lock the shared queue
    //            Pop the event data
    //         Unlock the shared queue
    //          Process the event
    //    If a request to terminate was received
    //        Terminate the worker thread
}

This is a pretty straightforward algorithm that should be easy to implement. Returning to the issue of file locks from earlier this algorithm introduces some issues however. Using the simple approach to locked files if a file is locked then the worker thread needs to stop processing any more events until the original event is processed or it times out. A local variable can be used to store the event. The variable needs to be checked before processing any events from the queue. Here is an updated algorithm with the changes in bold.

private void SomeWorkerThread ( )
{
    //While not terminating sleep for a fixed period of time
    //    If current is set
    //        Process the event
    //    If failed
    //        Increment current retry count
    //    If retry count exceeds maximum
    //        Report error
    //        Clear current
    //    Else
    //        Clear current
    // If current is empty
    //    While there are events in the queue
    //        Lock the shared queue
    //            Pop the event data into current
    //        Unlock the shared queue
    //        Process the event
    //        If failed
    //            Increment current retry count
    //        Else
    //            Clear current
    //    If a request to terminate was received
    //        If current is set
    //            Report error
    //        Terminate the worker thread
}

Enhancements

Access to the queue is serialized. This is overly restrictive given that the only time there will be a conflict is when FSW is trying to insert the first item into the queue while the worker is trying to retrieve it. A better synchronization approach may be useful. A bounded queue (MSDN Magazine, CLR Inside Out, May 2007) would solve the problem nicely and improve performance.

The second scalability issue involves the worker thread itself. Depending upon how many events are received the worker thread can easily get behind. Multiple worker threads would allow better scalability such that many events can be handled simultaneously. Ignoring locked files this is pretty straightforward to handle since multiple worker threads would access the same shared queue. However locked files make using multiple worker threads very complex. Some sort of handshaking must occur such that locked files are shared across multiple worker threads to ensure events are handled in order. A better approach might be to use a single worker thread but have the worker thread be state aware for each of the files being processed. Some events can be handled in parallel (even in the case of errors) provided the events do not work on the same set of files. This is actually a lot harder than it sounds because of the possibility of files being renamed but nevertheless would solve the problem nicely.

Caveats

A few final caveats are in order about FSW. You can not use FSW on read only media like CDs or DVDs. There really is not much benefit anyway. You also can not use FSW on removable media.

If Path does not exist then FSW will attempt to create it when EnableRaisingEvents is set. The FSW requires read access to Path otherwise an exception will occur.

Earlier it was stated that you could call UI methods in the event handlers without worrying about what thread you are on. That is not entirely true. If you drag and drop the component onto a form then it will work. However if you programmatically create the component then it is not. The SynchronizingObject property of FSW controls this. When you drag and drop the component onto a form the form is assigned to the SynchronizingObject property. This causes the event handlers to be raised on the UI thread. When you programmatically create the component you should manually set the SynchronizingObject property to the main form or another UI control so synchronization will occur automatically.

You might think that filtering will help alleviate the buffering problem but it will not. Filtering occurs in FSW. Therefore the event must still travel from the file system to FSW through the shared buffer. There is little you can do besides processing events as fast as you can to avoid buffering issues.

Using the FileSystemWatcher