Sharing AI secrets at Membership Excellence

Membership organisations build vast quantities of expert knowledge and insight over time but making it accessible - to both members and staff - isn’t easy. It was this challenge that led our client World Commerce and Contracting to develop an AI solution - ‘Ask Tim’.

Julian Davis, Chief Operating Officer at World CC joined our CEO Alex Skinner to discuss their AI journey at an industry insight seminar  - AI Prep: Data-Driven Groundwork for Membership Excellence - at MEMX 2024 on Thursday 14th November. Julian began by explaining that when he was first asked to look at AI, his reaction was ‘I don’t know what problem you want me to solve with it’.

 

However, they soon saw an opportunity to harness AI by using it to address complex contracting questions that their President Tim Cummins was frequently being asked by members. Tim would search the World CC archive, find answers and respond with supporting resources and recommendations. They realised that AI could do the same tasks quickly and efficiently - freeing Tim up to focus on other priorities.

Julian shared a video with the seminar audience that showed ‘Ask Tim’ in action. It not only provides a summary answer and source material but also links to specific points in resources - such as specific paragraphs or video time codes - where the answer has been taken from. The tool is now in beta testing with World CC’s member council.

Julian told attendees that a key learning from the development process with their AI partner was that the quality of their data - their knowledge - wasn’t good enough. Consistency is key in order for the AI to be able to learn and create answers. For example, if you ask the same question using different wording in member surveys it can’t compare like for like results. A highly-engaged audience had many questions for Julian about Ask Tim - see his answers below:

1. How easy is it to upload videos, articles, etc into the hub for Ask Tim to review the content?

It’s no more difficult to how you would load an article to your website’s knowledge hub. If anything it’s easier, as you don’t need to add any of the metadata - keywords, tags etc. The AI ingests it and understands the content without you having to do anything.

2. How many hours of Tim’s time with the data scientist has it taken to get to a stage that it has been useful?

Tim spent an hour, three mornings each week with our data scientists  and it took around a month in order to come up with a first draft version of Ask Tim. After that, we recognised that we didn’t want to rely on just one individual so we got another subject matter expert to work with them to spread the load. We’re still having 1 hour sessions, 3 times a week and the scientists do work on the tool in between them. The sessions are less around questioning Tim and more focused on demoing back what they’ve produced, refining the tool and coming up with new thoughts and ideas.

3. Do you expect that this can be used for general membership enquiries, aside from content search?

Yes, I think so. The key thing to remember is that you need to have the underlying documentation - the data and content that sits behind it. If there’s no documentation, there’s nothing to reference. For example, if you’ve got a core system that has a lot of well-documented release notes and functional guides, then you could upload these and the AI can learn from them.  

4. How best/should we use public LLMs trained with association knowledge-based intellectual property to deliver AI assistance?

It depends on the type of public Large Language Models (LLMs) you use. If you’re using a model that relies on public data sources - like ChatGPT, you can’t fully trust where that information has come from. However, there are LLMs out there that you can run locally and keep your information private. Google has one that allows you to upload 200 documents - letting you privately test how an LLM will work with your data.

5. How do you maintain the quality of AI responses? Do you still advise members to do a double verification?

Yes, there’s a higher level of trust in our own information but I think it’s good practice to always do your due diligence. We’re aiming to answer the question but what we’re doing is providing the source materials so that people can validate the answers themselves. We’ve set the tool up to have different styles of answering. We’ve got the Ask Tim style - which is a detailed one or two page answer, and we’ve got an executive summary - but a link to the specific point in the source material is always provided. 

Governance around the use of AI is holding back a lot of organisations back - they’re scared of using it. But as long as you’ve got clear guidance around what you do and that you’re not uploading sensitive or commercial IP there’s no reason why you can’t use it.

6. In the seminar Alex said he believes websites will be dead in 5 years and search dead in 3. Do you agree? If so, what do you think will replace them?

Websites are brochure ware - a shop front to attract people in - there will always be a need for that. However, from a membership perspective when someone becomes a member their needs are different - we shouldn’t be sending them back to the shop front. Their needs are different - they need a much more personalised member portal pointing them to resources that are relevant to their needs - it needs to be a completely different experience.

7. Do you incorporate your AI with blockchain technology?

No we don’t. Blockchain technology is used to ensure the integrity of the data that has been provided. For example, a lot of people use it for certifications. I can see that there could be synergies between blockchain and AI but we haven’t explored that area yet - it’s a good question!