Wednesday, 27 January 2010

Herding cats and counting sheep

Whilst away Church of England attendance figures for the year were released, triggering the usual ritual Groundhog Day for terminal decline, renaissance, or business as usual, depending almost entirely on the prejudices and agendas of the people doing the prognosticating. It doesn't seem to matter much what they actually say. Everybody, of course, is abslutely right from their own point of view, especially when it’s buttressed by hard figures.

As it happens, however, yesterday Kevin Ryan, diocesan ICT manager came yesterday to explain to Bucks area deans and lay chairs the use of Tableau, a new statistical tool we are making available for deanery planning. This enables people to drill down easily into local statitsics. As well as Kevin, we were lucky that one of our lay chairs is a professional in the field of marketing statistics, and was able to correct many basic errors of perception and interpretation among our group of 21 people. From this exercise we learn a few home truths:
  1. The interpretation of statistics for serious marketing purposes is a fne art. What you are looking for affects what you find, and all kinds of allowances have to be made, with local knowledge, before you can arrive at a real answer to pretty much any particular question. It can be done, but not without a lot of work and qualification. Look at two examples from Bucks of explosive growth, more than doubling numbers over five years. One is caused by the fact that it is a reopened former redundant church near new housing.The other is caused by significant Evanglism more than doubling the size of an already large congregsation in five years. Make what you will of either, but to make sense of data you need to consider contextual and local realities.

  2. Figures usually look enticingly accurate and absolute simply because they are figures, and only very occasionally because of the precision with which they describe particular realities. Go to any school governors' meeting in the league tables season, and you'll see what I mean.

  3. Everything depends on the accuracy of the data that went in to begin with. Church statistics, especially attendance ones go through three significantly wobbly layers of filtration which cannot be avoided... starting with the fact that they are collected largely by volunteers.

  4. Level 1: One is the significant differences in counting methods and consistency, which vary from never to regular, as well as from scrupulous to megasloppy. When I was an incumbent I once took a Sunday out to check our numbers. I physically counted 242 people on the premises. The number in the book was 127. This led to two significant discoveries about my parish. (a) the greatest variable factor was usually, in fact, which verger had done the counting. The more otpimistic ones tended to count higher, the more pessimistic lower, and some always the same kind of figure by a process of dead reckoning whereby they would go to the vestry, check the figure they'd put in last time, ask themselves whether the service felt fuller or not this week, and then stick something in the book. I’m not proud of this doscovery, but it was the way I discovered things were generally done. Accuracy was higher for smaller services simply because it was very much easier to count 16 people at evensong than 242 at the main morning service.

    Level 2: To derive the usual Sunday attendance figures, you need to average four Sundays in October noting any exceptions (like, say a baptism in the main service). This, along with the mathematical abilities of clergy and wardens, reveals a cornucopia of results, some more compliant with the theory of how they should be collected than others.

    Level 3: Returns rates vary. Only one deanery out of ten had an almost complete dataset and for another it was almost non-existent. Those who want to count sheep need first to herd cats.

  5. As a result of all these realities, the grossed up statistical picture may, surprisingly, be the most “accurate.” I don’t mean the figures are actually correct, they surely aren’t, but grossed out across a field through a few years, trends may well appear, with the inaccuracies cancelling each other out a bit. The big story seems to be one of significant and fundamental erosion in the nineties, and a generally flatlining pattern this century. It’s a good way of disappointing everyone with an axe to grind, but it seems to be the truth. It’s is probably the least promising straw from which to make bricks to load into coshes, but I'm sure noble attempts were made last week.

  6. The best use of these figures is local, because that's where people have the contextual knowledge to make sense of them. Thus our use of Tableau.

  7. These figures are like reported crime rates. Other oft-quoted data, for example, the English Church Census, is of the nature of the Home Office Crime Survey — the dataset behind it is very much more partial and any extrapolation from it very much more an expert game, like long range weather forecasting. Caveat Emptor.

  8. The figures we drilled down into, with some degree of local knowledge, revealed a huge variation, from the kind of classic decline patterns Fleet Street assumes, which do exist occasionally, to a far more egneral static pattern, and one or two examples of swift decline or explosive growth. More growth than devcline among these last. There are a handful of Bucks Churches which have experienced the kind of growth where they have had to double up shifts to fit everybody in, and one deanery in which, across almost thirty parishes, numbers have more than doubled over the past five years. What are you likely to experience in any of our 288 places of worship in Bucks? If you really want to know you’ll just have to go and see...


Archbeship Anthony said...

Hi All,

I am pleased to say that I live in and attend church in one of a small number of parishes where the congregation size is actually growING (bucking the national trend). I now attend an Anglican church even when away from home, Especially when I travel alone and for Important Sundays like All Saints, Remembrance, Palm Sunday, and Easter and Ascension. Others I cannot remember at the moment. I also attended Evensong in Hereford Cathedral was a really pleasent experience. I found myself signing the ending to most psalms without thinking about it.

Vinaigrette girl said...

Tableau looks easier to use than SPSS and SAS.

I suppose the question remains, though, as to whether you want your stats to enable you to ask (and answer) predictive questions as well as to analyse past data. I'd also wonder if the stats you do collect might be made (a) easier to collect (b) easier to standardise and (c) if some of the categories might be refined or recast.

Bishop Alan Wilson said...

To be really honest, VG, I'm not sure we've really got going with this stuff at all. If w ecan get the reporting rate up, it may be possible to do what you suggest, which i think would be helpful. A few people in our meeting made similar suggestions. The thing is the figures are collected on a national "statistics for mission" form, so the only way of changing it would be to work through General Synod, I suppose. In the meanwhile we're trying to make what sense of them we can and get as uch raw data in as possible. I'm so grateful we've got Wendy (Lay Chair who actually knows what she's doing with figures). We'll just have to see where it goes. One thing's for sure. Any statistics you see from the C of E have amazing degrees of variation and uncertainty locked up in them at every level.

Related Posts Plugin for WordPress, Blogger...