Evaluating new products for your farm

Farmers and their managers are inevitably in the firing line when salespeople call (usually at busy times) with products that "have got to be good for you" and claimed to increase profits – not just income. There seems to be no end to new products coming along which is good, provided you only buy those that work!

It is amazing how many farmers buy a product and apply it to the whole farm or treat all the animals so have no idea if any benefit was achieved. When asked, many claim that the grass is looking greener, and stock looking better assuming these are proof positive that what they bought has worked. How come that they never use it again?

You have to wonder how many new products are purchased just to get rid of the salesperson, and then don't want to admit that either the product has not worked or that they dare not admit their money was wasted!

Managers can always refer salespeople to someone further up the responsibility tree with the chequebook, but this trick may not work and the boss may ask you for comment and evaluation of a product before any decisions are made. Here are some points to be aware (and be wary) of.

"Research has shown"
Be very suspicious of this statement! Usually the salesperson will have been told this too, so won't know where the research was done or the details of the trials. It may have been published in a peer-reviewed scientific journal, in a farming magazine, a publication from the company making the product or just a press release. You really need to know which it is.

Research done overseas
You'll be expected to accept that "overseas research" must be good – rather like "overseas experts". But the concern is whether it has any relevance to your farm. For example under New Zealand's outdoor grazing systems with small cows run at high stocking rates, research done in countries with large grain-fed cows housed most of the year cannot be relevant – and vice versa. Or New Zealand's soils may be very different to overseas soils a product was tested on.

Research done on small numbers of animals & small plots
Research is a costly business so researchers trim the numbers of animals in trials and do tests on small plots to provide the least numbers from which a statistician analysing the data can get a "statistically significant result" (see below). Even if few animals and plots will satisfy a statistician, you must ask if the results will be important and make money on the farm – that's the real question you have to answer.

You can always be more confident if the research was done on a large group of animals and on large paddocks under conditions that were as much like a good commercial farm as possible.

Research that may not show statistical significance

The data analysis needs to show that the treatments in a trial produced a result that was not just due to chance. The statistician has to show at what level of probability the results happened. The three accepted levels are:

· "Significant": 5% probability which means that five times in 100 the results would have been due to chance.

· "Highly significant": 1% probability which means that once in 100 that it would have been due to chance.

· "Very highly significant": 0.1% probability which means once in 1000 that it would have happened by chance.

The problem is that the word "significant" is used as a general term so it's obvious that you need to know the level it applies to. For example 5% significance may not be enough and be far too risky to make any big financial investment.

Research with no economic evaluation
Very few trials employ economists to do sums about the commercial implications of the results. Researchers argue that as every farm is different, it's up to farmers to work out for themselves if the results would make money. This is a copout and there's no reason why some costs and returns cannot be done.

If you see any economic benefits mentioned – just make sure they don't just cover extra income from the product; make sure they include any extra costs as well.

Research without control treatments
This is a major deficiency, and you see it all the time. The excuse is often made that the researchers are trying to keep costs down so they only compare their own product with the opposition's product. Without a control, who is to know if their product was any better than doing nothing?

Short-term trials

This is another hazard where decisions are made over treatments that were applied only once or for only a few weeks or months so the long-term effects cannot be tested. In a biological world, changes take time and these can often be missed. Examples are low feeding levels of young animals before weaning having effects later on in life, or one treatment to a pasture which is only measured in the first grazing.

There's also the influence of seasons and it's dangerous to conclude much from a result from only one season. You really need to run the trials over at least two seasons.

Doing your own farm trials

You are often driven to this because you may only get half the story and little meaningful information from those selling the product. But be careful before you start, as it's never easy for a number of reasons.

· You'll need time to set up a trial.

· It will need a commitment by everyone on the farm - as may trials have been sabotaged by folk who may see it as extra work for which they are not being paid. Tags are lost, gates are left open and record books lost.

· There will be extra costs of things like tags, pegs, fencing and time to enter and process records. Making sure the data are backed up is important, as many a record book has been lost (or eaten) near the end of a trial.

· Avoiding stuffups! Examples are double tagging stock as tags get pulled out, putting extra hot wires around treated areas and putting extra catches on gates.

· Allocation of animals to treatments must be done at random, and treated areas of a paddock should be similar and avoid areas under trees and boggy areas.

· Make sure you have a control group or plots (untreated) and that these are chosen at random too.

· Have good big groups e.g. at least 20 cattle per group, 30-40 sheep, and paddock areas of 0.5ha as you'll have stock losses and lost data, and you need commercial sized groups and paddock areas to be confident that the treatment (s) can be seen at farm level.

· Unless the differences are large and very obvious, then conclude that tests were negative.

Get some advice
By all means do some testing of products on your own farm but consider these points the days of getting products tested by MAF for free are long gone! Ask a farm consultant for some help. There are plenty of statistics packages for PCs now to analyse results at very little cost. Spreadsheets make data recording so easy now so find someone who can do this for you. See if your neighbours would like to co-operate as this would give another environment to test the product in.