Bogus file corruption issues with TortoiseSVN on Windows 7 64-bit

Last Thursday I encountered a strange and frustrating issue. There are a wealth of blog entries out there about this, for anyone who bothers to search, but I am going to summarize it regardless. I myself didn't google it until after it happened to a coworker as well - at that point I knew something was up.

I've used Windows 7 64-bit with Microsoft Security Essentials at work for over a year now. For the last 4 months, I've had an Intel SSD in my workstation at work as well, and did not hit this issue until last week, when it hit hard.

I could not successfully checkout a new Subversion working copy to my SSD - period. Every attempt gave me the following message, in my case after 5-10 minutes time. Repeat attempts (update to attempt to resume the checkout) would cause it to happen again until finally the workspace could not be updated or cleaned.

Each time I would see the following error:

Error Can't move
Error '...\path\to\.svn\tmp\entries' to
Error '...\path\to\.svn\entries': The file or directory is corrupted and unreadable

Following this, the next reboot windows would schedule a CHKDSK on my SSD drive. Despite it not finding any issues, I still got a sinking feeling. Checkouts to my regular hard disk worked fine, aside from crawling in comparison. Hmm... As it turns out - my SSD did not have corruption problems. It also isn't TortoiseSVN's fault. It's a bug in Windows 7.


Fortunately, there is a secret agent style hotfix (email registration, download link with password that expires in a few days) that came out in June. It will apparently be included in Service Pack 1 when it is released.

Symptoms listed from the hotfix described exactly the behavior I was seeing:

For example, you run an application that uses the MoveFileEx() function to replace files. If you use this function more than one time in a short time, you may receive an "ERROR_FILE_CORRUPT" error message. Additionally, you may encounter one of the following issues:A warning message is displayed in the notification area. This warning message indicates that you should run the Chkdsk.exe utility to check the disk.
A disk check is scheduled for the next time that you start the computer.

Subversion uses the MoveFileEx() method for replacing files in place. The reason this didn't happen on my regular HDD, is that the issue is specifically when MoveFileEx() is called in fast succession. SSDs perform the actions must faster. They are apparently so fast, they uncover obscure filesystem implementation bugs in NTFS. Have I mentioned SSDs rock for development?

For the curious - file indexing and anti-virus could also apparently make this problem worse / more apparent, but disabling Antivirus did not help in my case. I didn't try disabling indexing, but I did not have whole-drive indexing enabled anyway. The hotfix completely resolved the behavior on my machine.

Here are some other reference links I found helpful. Hurrah for ServerFault, which had the first useful result in Google.

http://serverfault.com/questions/72561/64-bit-tortoisesvn-on-windows-7-says-file-or-directory-is-corrupted-and-unreadab

http://tinyurl.com/w7corrupt

It both amazes me, and ceases to surprise me that bugs like this shipped in a production OS. I mean...think of the madness one could go through thinking your HDD was corrupt, or software faulty, if you just believed the messages and didn't search. It is due to this that I post this entry, hopefully it will save at least one person a few wasted minutes.

This is also the second hard drive related issue I have encountered with Windows 7 so far. The other had to do with my 1TB hard drive disappearing after resuming from standby - it was something along the lines of Windows 7 not waiting long enough for drives to respond on wakeup, and larger drives can take longer to wake up and respond. I forget if I had to apply a hotfix, or update a driver or registry setting to fix that problem.

This isn't a rant about Microsoft, this is just another good example that writing software is HARD.

It's November again...

Last year I did my own take on http://www.nanowrimo.org/ by writing in this blog every day for a month. I plan on doing it again this year. Hopefully the writing habit will stick.

I intended to post this last week, but as it is already November 1st - I guess I am already behind.

I have quite a backlog of miscellaneous, semi-completed blog entries in the past month alone, so the first week may be completely random. I would like to come up with some topics that I can focus on in the weeks following - we'll see how it plays out.

There is a lot to talk about right now in the Java world.
- Where is Java headed? Feelings on the Java 7/8 roadmap
- Is Oracle scaring off developers, and what will happen to Android with Oracle vs Google?
- What is the future of Java on the Mac?

The web also continues to be full of interesting advancements, especially due to the iPad and mobile devices.
- Some very impressive things can be done with HTML5 + CSS3. What are some examples?
- Fierce web browser competition - it's hard to go wrong nowadays.
- Smart phone competition (Android, iOS, WebOS, Windows Phone 7, etc)

In addition to (hopefully) talking a bit about the above, and more, I'd like to lay out some other goals for the month:

- Start and finish reading Release It!, by Michael T. Nygard

I'll add more goals and topics as I think of them. Time to start writing Tuesday's post...

Ant DirectoryScanner memory problems with large directory trees

I encountered a very interesting build problem this week. Hudson running out of memory when trying to parse test reports. It wasn't actually even getting to the parsing part though. Thread dump looked innocent, Ant...hmm... so I took a heap dump and examined that a bit. Top memory use? Ant's DirectoryScanner. Ant's DirectoryScanner was running out of memory just trying to round up the test report XMLs. Even with a 1.3GB max heap, it was not sufficient to scan our workspace anymore. Peculiar.

From looking at the heap dump, DirectoryScanner holds on to unincluded directories. This doesn't seem right to me... I may dig further when I have time. Simple searches showed some archived mailing list discussions, but I don't think it's related. Ant 1.6.5, 1.7.0, 1.7.1, 1.8.1 all observed this behavior for me...so I do not think it is a regression. A simple ant goal run on this workspace - pure ant, couldn't scan the dir until it had a 1400m heap, and to do so took it 96 minutes because it was still memory constrained.

The pattern being used was fairly innocent looking:
koko/dev/projects/**/target/test-reports/TEST*.xml

Except it isn't that innocent when the root workspace dir has 300k+ files, and 1,000,000+ directories in it, with some very long path names on top of it. As it turns out, that ** is expensive and should be used with great care. Even though it is only traversing our code projects, it is still a very large directory tree and it runs out of memory.

More important than the workspace dir count, which should be getting skipped with the start of the pattern, the dir count within /projects/ is still almost 300k dirs.

Maybe this is a newbie mistake with Ant...but I'm not so sure. There are times where ** may be necessary, and I have a newfound awareness of how expensive this can be (in time taken, obviously, but more importantly in memory - it's no good if it fails because it runs out of heap...)



For a smaller directory tree on my dev machine, I wrote a short ant target that would touch 1 file, found using an include pattern similar to the above. The ** version took 24 seconds to execute on an Intel x25-m 160GB (G2) SSD. the * version took 0 seconds. The time penalty of ** would be much larger on slower, traditional hard drives...

Bear in mind, for Hudson users - Findbugs, Warnings, and other plugins typically use Ant include patterns (and I am sure Ant's DirectoryScanner), and in those cases sometimes ** is unavoidable.

I need to fire up a Hudson workspace and ensure that it supports multiple includes, so users (me!) can avoid ** in more cases...

Don't Filter Without Being Aware of the Scunthorpe Problem

I just happened to stumble across this question today while browsing StackOverflow. In it, a user is searching for a library that will filter profanity. Some suggestions are posed - as well as a problem with one of them. The Scunthorpe Problem. The gist of it is that naive filtering implementations will often filter innocent words that contain sequences of letters that are considered dirty by themselves.

I highly recommend reading the full wikipedia page - it is highly entertaining.

If you ever find you need to implement a profanity filter - make sure you don't forget about the Scunthorpe problem, or you may cause your users unnecessary strife.

My favorite word resulting from the scunthorpe problem: buttbuttinate.

More on e-reading, and the Pragmatic Bookshelf

I had some ranting the other day on my first experimentation with e-reading, but afterwards I learned something cool to offset the potential downer. I had completely forgotten that I have PDF versions of lots of books from the Pragmatic Bookshelf.

I love the Pragmatic Bookshelf. Not only is it full of awesome books, but books I bought 5 years ago in print+pdf I can now convert and put on my iPad at no additional charge. This is the definition of sweet. All I had to do is log into my account, tell the gerbils which books I wanted in what formats, and waited a minute. There are a couple of old ones that are still only available in PDF - but I can still view those on my iPad, it just isn't as convenient yet. The Pickaxe is a notable example...but I'd say more than 75% of my books had e-reader formats available. Again, iOS 4 for iPad can't come soon enough - in this case for iBooks PDF reading.

At a bare minimum, I know that I can enjoy a good set of books on my iPad going forward. Exciting. Especially since there are still a couple I haven't read yet, and some I intend to re-read soon. There are countless more that I desire...

It's high time I create a bookshelf on this site, come to think of it... I'll put one up in the next week.

Adventures in E-reading

I read my first book and a half on the Kindle reader for iPad last weekend.

Specifically I enjoyed:

• Masters of Doom - a great book primary about John Carmack and John Romero. If you are a fan of the genre, it's a great read. I stayed up past 4am reading, it was so engrossing. I remember being on the Software Creations BBS, which is mentioned in the book.
• Peopleware - Productive Projects and Teams (Second Edition) - a highly recommended book. I'm only half way done, so a full write up will have to wait, but it is a very insightful (and also quick) read. 

Why did I choose the Kindle reader over iBooks? Kindle has a much better selection of books. So far iBooks is 0 for 3 on books I have wanted. Also - the Kindle reader is available on other platforms.

I love reading on the iPad. I can read with a light on - or without. The text was very readable and my eyes did not get put off by the backlit screen even after reading for 5 hours straight as I tore through Master of Doom.  Even though I already owned Peopleware in print, I paid the 10 bucks for the Kindle version anyway. That's a testament to how much I enjoyed reading on the iPad.

All is not well in the land of e-reading though. While I had no quality issues with Masters of Doom, Peopleware seems to be full of missing punctuation, at least one blatant typo, and a chapter that is in the wrong place. I did a quick search - Apparently I  am not  the  only one. Since I have the print version, I have verified that none of the issues I have seen occur in my print copy. What is going on? Could it be that physical books are being OCR'd, or worse, re-typed by hand? What on earth??? Shouldn't digital copies already exist at the publisher?

Here is an example from the Table of Contents in Peopleware:

Actual TOC from the Paperback copy

Kindle version

The chapter appears in the wrong place in the book. Intermezzo should be between chapters 9 and 10, not between chapters 8 and 9. Also, less importantly, the titles seem to have been truncated, and in the case of #8, the quotes are missing.

I found at least several instances of missing punctuation while reading so far, as well as one glaring typo:

"The only acceptable interruption there was a fire alarm, and it had to be for a real Tire."

Somehow a lowercase f turned into a capital T...

Maybe some more veteran e-readers can tell me if they run into this a lot. I find it seriously distracting when sentences are incomplete or I find typos in books. Especially if they are an artifact of the e-book translation, and not something the original editors missed.

Is this the state of ebooks these days? I hope not, or my adventures in e-reading will be short lived.

Stage. Stage. Stage. Always Stage.

It doesn't matter what software we're talking about.

• MySQL
• Hudson
• Java releases
• thirdparty libraries
• Subversion client
• you name it - if it can be upgraded or replaced, it can and needs to be staged.

 Always stage. ALWAYS.

If you don't stage, you're just punishing yourself (and potentially your coworkers and/or users). So be smart - always stage your upgrades. You'll get burned if you don't. It's just Murphy's Law applied to technology, really.

This isn't in response to recently getting burned myself, it's been a post that's been brewing for a while.

Many years ago, a friend of mine once wondered why an admin wouldn't upgrade PINE (a popular terminal email client at the time) as soon as it came out. Now I know why - because that admin was wise. They wanted to be sure it worked and didn't cause any regressions before subjecting their users to it.

I am continually surprised not just by how many things can go wrong, but also how frequently they do, despite the best efforts and intentions of developers. Just the other week, the tinest name change by Oracle in JDK 1.6.0 Update 21 caused Eclipse to fail to launch. Who'd have thought? And yet it happened. There are numerous other examples. I am sure you can think of plenty in whatever software you use. I can certainly think of several, just in the past couple of months.

Before your whole office upgrades to the latest version of Visual Studio, or the latest Subversion client, or any other software. Make sure someone tests it out first.

Stage, because anything that can go wrong will go wrong.