3 engagement metrics that look important but aren't (and what to track instead)
A mistake some organisations make when building an Engagement Scoring Model is starting with the metrics that are easiest to collect, rather than the ones most likely to predict the outcomes they actually care about.
The data your system produces tends to be available in abundance and it feels meaningful. But when you sit it alongside actual renewal rates and revenue data, several of the most widely used signals turn out to be telling you far less than you assumed.
Before you weight a single metric, it is worth asking what you want your engagement score to do. There are many ways to use engagement scores. Do you want to use it to increase retention? Or to find parallels between the types of engagement that lead to event bookings?
Consider a member who rarely logs in, but attends your conference every year and presents at it:
- Under a retention objective, they might look Orange “at risk”, with low engagement.
- Under an advocacy objective, they might look Green “highly engaged”. That same data can then feed a deliberate programme. You might invite this member into a formal referral or ambassador scheme, or ask them to take part in a case study or a podcast.
What happens when you measure the right things
Before one of our clients implemented engagement scoring, their team had visibility across three member behaviours: events, room hire, and training. Working from those three signals, they estimated that around half their membership was inactive.
Once the model was running across more indicators, the picture shifted substantially and the proportion of genuinely inactive members was closer to 10%.
The difference was not that members had suddenly become more engaged. It was that the team had been measuring the wrong things, and some of the behaviours that felt like useful proxies for engagement turned out not to be.
The three metrics below appear regularly in membership engagement models. None of them is useless in every context, but all of them tend to be overweighted relative to what they actually predict and the result is a model that looks rigorous but directs attention in the wrong places.
What metrics to take caution with
1. Email open rates
The intuition behind including Email Opens is reasonable as a member who opens your emails is at least seeing your communications. The difficulty is that open rates often capture something about inbox habits rather than genuine engagement. It might be that some of your least committed members will open everything reflexively, while some of your most loyal members are simply protective of their time and open very little.
The reliability of open rate data has also degraded significantly since Apple's Mail Privacy Protection began pre-loading email content regardless of whether the recipient actually engaged with it, inflating open rates across a large portion of most membership databases.
A more useful signal: The Engagement Scoring Module lets you track whether a member downloaded a resource or attended an event that the email promoted which are actions that reflect genuine engagement, not inbox habits.
2. Event attendance without context
Our Engagement Scoring Module includes "Attended an event" as a metric by default, but it is important to remember the context of the member journey and the stage that person is at.
For example, an undergraduate student booking an event with your association for the first time may represent a more meaningful engagement than a fellow who attends events as part of their work. Looking at the stage each member has reached will help you analyse the data more accurately, and from there you can adjust the weighting for specific tasks within specific sub-groups of your members at every stage of their journey.
A more useful signal: Combine event attendance with other data about the stage of the member's journey. You may also want to include post-event survey data to check they actually found the event useful and it left a positive impression. Working only with behavioural data can be reactive, whereas attitudinal data can be predictive. A member can be behaviourally quiet but attitudinally loyal (they rarely log in, but they rate you highly and share widely). Behaviour alone would flag them as a churn risk, attitudinal data gives you the whole picture.
3. Being added to a group
Being added to a ReadyMembership Group is different to joining a group of your own accord. One is an administrative action the other is a choice the member has made.
This distinction matters because group membership is an easy signal to inflate without meaning to. Many organisations add members to groups in bulk and every one of those members then appears to carry a "belongs to a group" engagement signal. In reality, a proportion of them have never opened the group, read a document, or spoken to another member. If your scoring model awards the same points for being placed in a group as it does for actively choosing one, you are rewarding your own administrative processes rather than member behaviour.
The signal you actually want is participation, not presence. A member who was auto-added to a regional group and never interacts with it is telling you something quite different from a member who sought out a special interest group, joined it themselves and contributed to it.
It is also worth watching the signal in the other direction. Leaving a group is a deliberate act too, and depending on your objectives it can be a meaningful early indicator. A member actively pruning their involvement may be quietly disengaging long before it shows up in renewal data. The OOB Engagement Scoring Module includes both "Join group" and "Leave group" as metrics, so you can weight the choice to join positively while treating a departure as a flag worth investigating rather than ignoring.
A more useful signal: Are they sharing Group documents? Are they attending Group meetings? If you have a Discourse integration, are they participating in your forums there? These are the behaviours that distinguish a member who is genuinely part of a community from one who just was added to a list.
