Before my current rails project i was doing good ol’fashioned development in Java :-)
develop a feature, write unit tests where possible, and then play about with the whole application to see if it integrates. Then over to the test team who did user acceptance and integration tests.

During development a build server is an absolute must.

Now, in my experience you still need users (user acceptance) - and if possible professional testers (exploratory testing) to have a go, because as a programmer you have a bias that will keep you from making the application foolproof.

Performance tests are a requirement for high-volume applications - so if that’s the idea than it’s good to plan for it soon. However, some applications will take some time acquiring critical mass anyway, so it could be worked into later iterations.

Finally, it’s good to be able to rehearse a deployment into production.
That’s about all the testing i’ve done so far, seems so work.

@Jim: Thanks for all the great tips. I like your recommendation of 1 year plan, with incremental build-up. Realistically, it seems that many of us need to: (1) declare test bankruptcy, (2) get rid of the binary mode of thinking of ‘either you’re there, or you’re not’, (3) start building up the test suites over time.

Unfortunately Rails-land does not yet have a clean CI application. There have been a few plugins released in the past, that auto-run your test suites, but there is definitely a lot of room for improvement in this area.
I guess at the end of the day, we just have to remember that testing does not disprove presence of bugs or unwanted behaviours – hence, while we should strive for 100% code coverage, you also have to be prudent with your time. ;)

@Simon: “Anyway, why waste a lot of time writing test cases when I’m in such a rush to release product?” Believe me, that’s been my argument for a very long time… And it’s true up to a certain point in the lifecycle of the app. Eventually you’ll hit a wall, where small modifications will result in cascading effects of (weird) behaviours, and that’s where a good testing framework will save you.

@Jesse: 500 internal server error on your blog, doh?

—

A lot of people have followed up via email – it looks like we’re not comfortable discussing these things in public. Here is a brief (anonymized) breakdown of the responses:

Full RSpec (or equivalent coverage): 0 out of 18
Extensive unit-testing suite: 5 out of 18
Ad-hoc unit-testing suite: 8 out of 18
Manual testing, prior to release: 8 out of 18 (rest, either automated, or cross-your fingers)

Granularity:
- low-level unit tests for all components: 4 out of 18
- functional level: 8 out of 18

An interesting result (arguably not surprising) is that the test-coverage / practices appear to be directly correlated to the size of the company. More hands on the code, larger code-bases, all surely contribute to this trend. The trick is in finding that happy medium of agility when you’re a one/several man shop, and going light on the testing, and then eventually instilling the practice of test integration before it becomes a major pain in your organization.

http://whoyoucallingajesse.com/past/2008/4/9/release_and_testing_procedures_in/

I didn’t want to do what Jim did ;)

UI testing - Selenium [http://selenium.openqa.org/]
Unit testing - Sort of SimpleTest from CakePHP [http://simpletest.org/]
Perf testing - is between Pylot [http://www.pylot.org/] and Apache benchmarking [http://httpd.apache.org/docs/2.2/programs/ab.html]

My next improvement is going to come from improving the release pipeline. I want a system that can move data from dev to test to production automatically with a button push.

Once I get that started I’ll doc it here:
http://www.janisb.com/blog

Anyway, my approach to testing is rather haphazard. I guess that at the moment, I start writing unit tests when I hit a bug that I can’t fix quickly some other way. Anyway, why waste a lot of time writing test cases when I’m in such a rush to release product?

In order to avoid chaos I try to develop software in as clearly divided units as possible. So, given the choice between trying to integrate some library into Rails (for example) or just running it as a separate server and using HTTP I’ll choose separate server. That also makes it alot easier to hand-test, as I think a lot of the power in testing is simply how accessible is the code to be hand-tested or isolated anyway. And if something’s more isolated it’s easier to hand test (and notice problems in any case).

As for deployment, I try to make it as simple to deploy as possible, for example don’t need root to do this, one-liners, etc. And obvious if it’s broken. Then it’s a simple matter to automate it with scripts when it’s necessary. But I haven’t done any really complicated server deployment.

Well, that’s all I have to say for now. I probably fall fairly far into the laissez faire side of the spectrum, but at least I have a fairly good idea of what the options are :-)