Significant Science

Before we begin, Jon, I’d like to give readers a bit of background on how I came to learn about the TRIP Database and why they should know about it. I envision the readers of this interview as most anyone in medicine and healthcare who needs to search for medical information as well as non-medical people who want to go beyond the world of Google when searching for health information. In these days of the e-patient/empowered patient, the latter is a growing group.

You and I talked on the phone and this interview is a follow on to that chat.

I have worked in a medical library, but I must confess that although I had heard of the TRIP Database, I had not used it in my job. Given the importance of the evidence-based medicine movement, I appear to have been quite remiss in not utilizing the TRIP Database.

Let’s start with the basic terminology of medical search. You use the word “database” in describing TRIP. Now, like most people in the medical library field, my mainstay tool was PubMed, which uses this wording, “The PubMed database comprises more than 19 million citations…” And yet when I did a Google search on the term “PubMed” I also saw such wording as, “PubMed is a free search engine,” “an online US Government index,” “an online library” and so on.

And on the TRIP Database homepage we see the wording, “TRIP Database – Clinical Search Engine…The TRIP Database is a clinical search tool designed to allow health professionals to rapidly identify the highest quality clinical evidence for clinical practice.”

Could you please delineate for us the differences between a search engine, a search tool and an online database and provide examples of each? For example, how would you categorize the grants and scholarship listing I work on ScanGrants? It is a search engine, a database or both? I helped create it and I am not sure what it is! Help!

Unfortunately, I’m not the best person to ask! I imagine there is much overlap between them all and in many ways they could be viewed as synonyms. If they are not strict synonyms, I imagine people use the terms in such a manner.

What does TRIP stand for?

It initially stood for ‘Turning Research Into Practice’ but we dropped that many years ago, simply using the term TRIP.

Could you tell us about your own background and that of your co-founder and co-director, Dr. Chris Price? I note that you started TRIP in 1997. What was the impetus for the creation of TRIP? Was there a defining moment or did you and Dr. Price just feel that clinicians lacked access to the information they needed to provide topnotch care to patients? Could you elaborate on this, “…a chance conversation led to the spreadsheet being converted to a crude web-based search engine?”

I’m a scientist by training and Dr Price is a general practitioner (similar to the American Family Physician). I ended up working for the NHS and was asked to try and improve the uptake of evidence. I spoke to a large number of health professionals and they clearly said they wanted us to answer their clinical questions and not to send them on training in appraisal and searching – they didn’t have enough time. It was as a result of that I started the ATTRACT (Ask Trip To Rapidly Alleviate Confused Thoughts) service which is still going strong. As part of the process of answering the clinical questions I found I went from site to site and each site visit added time to the answering process, so I decided to make a list of all the documents in an excel spreadsheet; recording the title, URL and date. At the same time I was thinking of starting a website for ATTRACT and had a conversation with a man in the office opposite, who was an early web enthusiast. He said he could make the spreadsheet searchable – and he did. The same reason we started the TRIP Database (making clinical Q&A easier) is still going strong and we’re still heavily involved in Q&A (see www.tripanswers.org). We’re even thinking of starting a journal of clinical Q&A.

Can you tell us in what circumstances a PubMed or Medline search are just are not enough? And while you are at it, would you please tell us the difference between PubMed and MEDLINE and how TRIP differs from each?

I think there are important differences. PubMed/Medline are massive databases of – mainly – primary research literature (clinical trials etc). In the world of evidence, primary research is deemed to be less strong/robust compared with secondary evidence. Also, most people searching today are used to searching Google. That approach won’t work with PubMed. TRIP differs in that we allow users to search across a large body of secondary research, primary research and etextbooks. Our aim is to allow users to find answers to their question using the best available evidence. If there is no robust secondary research we’ll return primary research. Also, our aim is to make the search similar to that seen in Google. So, in essence we want to make it as easy as possible for clinicians to find the most robust evidence for their search query.

You say on the TRIP site the following, “The evolution of the TRIP Database has been guided by the desire to answer real clinical questions using the principles of evidence based medicine.” I want to parse that sentence so that I can grasp matters better.

For example, could you give us two real-world examples of “real clinical questions” and why a physician would use TRIP to answer them? For instance, in what scenarios would she be better advised to use PubMed only, TRIP only, or TRIP first or PubMed first? What other online resources (free or otherwise) might she consult in addition to TRIP and PubMed? Do you have any special advice for American users of TRIP (given that you are UK-based)?

Lots to discuss there! I’ve got experience (personally and my teams) in answering over 10,000 real clinical questions. I would always start with a TRIP search and that’s a firm part of the various Q&A methodologies. Say the question was ‘Are statins useful in the elderly?’. A search of TRIP immediately returns a number of high quality, secondary reviews on the topic. Searching PubMed with the obvious search phrase ‘statins elderly’ (as most non-librarians would attempt) yields pretty unhelpful results.

Another example question might be ‘Is vitamin d useful in osteoporosis?’ Compare TRIP’s results with PubMed’s . I’d like to think the results speak for themselves. TRIP returns high-quality, clinically useful articles while PubMed returns an awful lot of results which are unlikely to be clinically useful!

As for other resources, that depends on the type of question. If it’s about a drug side-effect or interaction I might use specialist sites (e.g. drugs.com or http://emc.medicines.org.uk), if it’s really unusual I find Google and Google Scholar can be invaluable.

You highlight that we’re based in the UK, around 70% of our usage is from outside the UK and our content is not tied to the UK, far from it. We want good quality evidence from wherever it has been produced. You’ll see from the site that we have 5 separate geographic areas for our guidelines: USA, Canada, UK, Australia & New Zealand and ‘Other.’

Let’s talk about evidence-based medicine. In the age of Obama, we hear a lot about the US government’s drive for “comparative effectiveness” to be taken into account in the practice of medicine. Could you please tell us how comparative effectiveness differs from evidence-based medicine or are they, in effect, synonymous? Do you forecast the Obama administration’s emphasis on comparative effectiveness increasing utilization of TRIP by American medical providers?

As I mentioned above I’m not the best person to ask about definitions. They may not be strict synonyms, but here is the definition of comparative effectiveness in Wikipedia :

“Comparative Effectiveness Research (CER) is the direct comparison of existing health care interventions to determine which work best for which patients and which pose the greatest benefits and harms. The core question of comparative effectiveness research is which treatment works best, for whom, and under what circumstances.”

That sounds very similar to the definition of EBM.

As for increasing traffic, it might well do so. As it happens, the USA already counts as our top country for traffic (tied with the UK).

You say on your site, “TRIP remains the internet’s leading resource for evidence based medicine.” What are other such resources? Please list some as well as their strengths and weaknesses. Have any of them, for instance, come along in the last year or so? And do they identify themselves as search engines, as databases or as both?

There are lots of search engines that have come along in the recent years. The barrier to entry is quiet low (compared to 5+ years ago). To be honest I don’t spend too long on these other sites, so am not really in a position to comment.

You also say, “TRIP have recently created an advisory board with wide representation.” Who is on it and from what fields and nations do they come?

I hope this doesn’t appear too evasive, but I’d prefer not to name names. Principally as we’ve never really discussed terms of reference and I’m not sure if I have people’s permission to name them! However, I would say that the vast majority are not ‘big’ names that people would recognize. We just have a group of around 70 people who I can turn to seek advice on various issues. There’s an even split between health professionals and information specialists. Geographically, they come from all over the world e.g. UK, USA, Mexico, Argentina, NZ, Australia, Germany, Spain, Peru.

Let’s talk about user feedback. You say, “We receive, and welcome, feedback from users asking about features they’d like to see…” and I was pleased that during our phone conversation you listened to my suggestion that users of TRIP be able to set their search parameters by date, given that users want to be able to search for items dating, say, from the last two years, last year and so on. You said that that feature is in the works for the next upgrade. Could you tell us when that upgrade will be announced and what we might look for in it? Can you give examples of who wants what upgrades and how you prioritize such requests? For example, do certain medical specialties want certain features? Do medical librarians ask for such and such whereas nurses (are they big TRIP users?) want this and that?

The main inputs for change come from two main places. Firstly, there are ideas I generate myself and these tend to come from using the site to answer questions or seeing what mainstream search engines are doing. Secondly, feedback from users. This latter group is essential as it gives a different perspective – so we love hearing feedback/constructive criticism.

As for the next upgrade the major change will be the introduction of ‘Related articles’ technology. I love the related articles feature in PubMed, it semantically analyses a document and finds other documents that are semantically close. We’re experimenting with a similar approach. However, we will not release a ‘related articles’ feature as such (it’s not as relevant on TRIP as PubMed). Instead, we plan to use it in the following ways:

• Natural language searching of TRIP
• Ability to paste in sections of text from anywhere and find close matches within our database
• ‘Follow this’. If you like an article, follow it, and we’ll e-mail you when a similar article is added to the site.
• Updating Q&As. We have over 6,000 Q&As; by comparing old Q&As to new content we can highlight new research relevant to the Q&As.

In addition we’re re-writing TRIP Answers and releasing a lot of self-test education based on our repository of Q&As. This latter feature allows a user to select any question, record what they think the answer is, we then show them our answer and then they comment. This sort of learning (reflective learning) is deemed highly educational and is a component of education in the UK setting.

Can you tell us how TRIP is financed? You say, “We have an exclusive advertising deal with eHealthcare Solutions.” The ads on TRIP seem unobtrusive. Is advertising your primary source of income or do you receive support from the UK’s National Health Service? From foundations? Are you an employee of the NHS?

TRIP is trying to diversify its income streams and have been moderately successful in this aspect. However, we always have more ideas than money, which is frustrating. But the main income streams are as follows:

• Advertising. I’m glad you found them inobtrusive.
• Web-services. We work with a number of groups to allow them to search TRIP via a ‘back door’. They send a search to us, we provide the response via XML and they seamlessly incorporate the results in their product, they can even control how the results look and feel. Examples of this include an electronic health record and as a data source for a portal’s search engine.
• Q&A, we run one small-scale Q&A service for a portal site in the UK.
• New research identification. We gather a lot of new research every month and this is useful to a number of people. So we have arrangements to supply new content in particular areas to various organizations.
• Bespoke search engines. Recently, we’ve been approached by a couple of groups asking for us to make them a search engine and we’ll be rolling out the 2nd of these shortly.
• Reviews of a given clinical topic. Not so much systematic reviews but more in-depth than a rapid Q&A.
• Website reviews. We get asked occasionally to review sites to ensure they’re fit for purpose.

As for myself, I still work 2 days per week for the NHS.

One feature I found interesting in TRIP is your openness to other search tools. For example, I tried in TRIP my favorite search term, amyotrophic lateral sclerosis, and saw at the bottom of the page, “Not found what you’re looking for? TRIP might not have the answer, but these other sites may…” and you list SumSearch and Google. I am not familiar with SumSearch. Please tell us a little about it.

I’m very keen for users to find what they’re looking for. I believe it’s arrogant to believe that my site (or indeed any site) will answer all the questions users have. So, why not plug in some tools to help users if TRIP ‘fails’. I’ve known Bob Badgett (the associate professor responsible for SumSearch) for a number of years. We’re both interested in the same thing and have approached the problem it in different ways. Bob’s approach is to use contingency searching in that he’s automatically tweak searches to return only a modest number of results. I’m not an expert of SumSearch so best to look at the site for yourselves and read this ‘proper’ explanation.

As for incorporating additional search features. I have no plans at present, but if a particular tool is highlighted that can bring significant extra benefit to TRIP (well, TRIP users) then I’d be keen to add it.

And given your willingness to feature other search tools (like SumSearch) does TRIP have any plans to partner with other health-related or research search tools such as Mednar or DeepDyve?

And speaking of DeepDyve, they have some pretty neat widgets and you offer one of your own:

Or would you even classify that as a widget? Indeed, let’s talk about the adoption of widgets in the healthcare industry and in the world of medical libraries. Many medical libraries are setting up intranet-based pages at their institutions as well as public Web sites for the use of those of their patrons who happen that point to be outside the firewalled institutional electronic environment and who are always offsite. Many medical libraries are also setting up sites primarily designed for use by the general public. And it is not just medical libraries that are doing so, but marketing departments at community hospitals or major medical institutions as well. Could you tell us who has placed a TRIP search box on their sites and who might benefit by doing so? For example, there are quite a few bloggers on the subject of medicine and health. Would they regard your widget as a nifty freebie offering to visitors to their sites? How would that work, exactly?

We don’t keep track of who has the widget! An example of it being used can be seen here.

I think that the adding of widgets is a great way of making a site more useful and sticky. By plugging in the TRIP widget you add a powerful search tool to the site. Basically, it’s a great free feature that enhances the site.

As for installation, it’s relatively straightforward (and we can advise) in that we supply a few lines of HTML code and they user needs to add that to their site. This is very easy for blogs but a bit more complex for traditional websites.

Let’s talk about the rise of social networking and social media and how they have overshadowed the once hot topic of search. How is TRIP adapting to and leveraging the power of social computing? Could you tell us about doc2doc, for instance and your relationship with the BMJ Group (itself a fascinating example of a mainstream medical publishing firm embracing the power of the Web for the dissemination of its offerings and engaging users fruitfully).

It’s an interesting topic and the future in medicine is difficult (for me) to predict. Our particular doc2doc tie-in will be launched properly in the near future. Again, it comes back to our desire to help users find the information they need. If TRIP fails they can try SumSearch, but they’ll also (if they’re a health professional) be able to ask clinical colleagues on doc2doc if they can help. We’re hoping the relationship will be mutually beneficial.

Aside from doc2doc we’re starting to examine the potential for a social aspect to TRIP. This has been fuelled by the large numbers of people signing-up to My-TRIP (a free feature that gives extra benefits to users e.g. auto-searching, recording activity etc.). We’re currently at 1,300+ after just two weeks. Each person records a profession, a country and clinical interests. As I see these people joining I see connections. These connections are either professional (e.g. nurse to nurse), interests (e.g. oncology to oncology) or geographic (e.g. USA to USA). I think we could do some interesting things in this area. However, the lack of funds will mean we can’t rush into this space. But, I (supported by the advisory board and others) will continue to work on ideas and see what happens.

And still on the subject of Web 2.0, I notice that under Sources searched by TRIP you list wikis, Webcasts and podcasts. (But not blogs so far?) Is that something unique to TRIP in the world of medical search and what do you see as the promise and challenges for databases of such new genres as online video journals like the Journal of Visualized Experiments (JoVE)?

Actually, we now search three blogs and we’re pleased to have taken the step. I’m sure we’ll add new blogs in the near future.

As for videos, our system can handle them and we had planned to roll out a video search. Unfortunately, we do not have the resource at this stage. That aside, the challenge with video search is for a user to locate an appropriate video. The only searchable aspect is the title which is problematic. For example the title of a video might be ‘Pharmaceutical approaches to diabetes’ and this might include a significant discussion on metformin. A user searching for metformin and diabetes would not find that video!

You include images among your search results. Is that also something that sets you apart from other medical databases? Where do you get those images and how do you determine their worth for your users?

Our medical images are a work in progress! It certainly does separate us from other clinical search tools. Medicine can be very visual, most obviously in dermatology. In the case of dermatology, if you’re unsure of a diagnosis it’s helpful to be able to see examples of things like rashes! Aside from dermatology, it can be useful to see examples of CT-scans, x-rays etc.

We created a bespoke spidering system to grab content from medical image sites that we’re aware of e.g. hardin.md. Currently, we’ve got a lot of images, not all of which are appropriate (e.g. they may be part of a banner) so we’re gradually editing the image collection to improve the quality.

Do you think that in some ways, given the rise of Open Science and the microcontent and preprint cultures, publication in journals as we now know them is becoming almost an afterthought or even unnecessary? How do you see projects such as Elsevier’s Article of the Future affecting the world of medical search and clinical research generally? Will such things leave frontline clinicians mostly untouched or will even a family doctor have to cope with the fact that articles will become fluid, ever-changing entities?

Unfortunately, this is a topic I’m not overly familiar with. Knowledge in medicine is already fluid, in that new research, advice affects the knowledge base on a regular basis. One issue is the speed this diffuses to clinicians. Will this improve things, I’ve no idea!

When we spoke on the phone, you spoke quite passionately and feelingly about TRIP’s relationship with HIFA2015 and mentioned a rather fascinating idea you are toying with of offering a button on TRIP that would enable those conducting searches to fairly painlessly add an article to a database of medical literature specifically designed to be of use to those providing medical care in resource-poor settings, specifically in the developing world. Now, that is an excellent idea in many ways (although I would say that when I do a search I really don’t have the time to think about poor countries and what their particular needs and resources might be—I am just trying to get my work done in an industrialized country and help those in my immediate orbit). Is there a term for this sort of thing? Altruistic searching, maybe? Has it been adopted anywhere else?

I’m excited by this idea as – if it works – it’s an easy model for others to adopt. Basically, we want to crowdsource the identification of content suitable for resource poor settings. TRIP has lots of such content, but it is ‘hidden’ by the ‘noise’ of articles based on setting such as the UK and USA. So, a user from the developing world (with bandwidth issues) may need to wade through content which is good evidence, but not applicable in their setting. If even a small number of users start ‘tagging’ content suitable for such settings it’ll allow users to restrict their search to content ‘suitable for resource poor settings’ – making a very focused search. The old adage about user participation on the web is that 90% of people will not interact with a site, 9% will do so occasionally while 1% will do so actively. We currently get around 35,000 visits per day, so 1% is 350 people per day potentially tagging articles. It’s certainly not for everyone, but the cause is ‘just’ so I think people will be inclined to get involved.

We have received funding to get this in place by the UK-based BUPA Giving scheme, so we hope to roll the feature out in early 2010.

As for a name, altruistic searching might work, a quick Google shows terms such as Crowdsourcing Philanthropy and Crowdsourcing Altruism! The notion of crowdsourcing altruism was seen when Steve Fossett went missing. In this situation very recent maps of the area where he may have disappeared were added to the web in bite-sized pieces (I think it equated to 10m x 10m of land) and users were asked to look for plane wreckage and click if the area was ‘clear’ or not. It never found the plane (as far as I’m aware) but did locate a number of other planes that had gone missing over the years!

What is CPD and what is its value?

CPD is continuing professional development. It’s dawning on me that this is a very UK-focussed term and I should use a different term. In the UK (and other countries) there is a requirement for health professionals to stay up to date with the latest evidence, this is a key component of CPD. It’s value is that if a clinician can’t demonstrate they’re keeping up to date it could affect their ability to practice. TRIP has released a way for clinicians to easily record what they’re looking at and also to assign reflective notes to a particular article. These reflections are aided by three guide questions:

• Why did you look at this article
• What did you learn from this article
• How will you apply this learning in practice

These can be captured in TRIP and over time you build up a portfolio of learning.

We’d like to use this approach for the USA but it’s an area I struggle to understand. In other words, in the USA is this approach deemed appropriate CME!?

Let’s talk Twitter. You say on your blog:

“We’ll be working on a mechanism to link TRIP in with Twitter.” How would that work?

This may or may not go ahead. Basically, each month we add lots of great content onto TRIP. So we thought we could drop topic specific content as a twitter feed. In my mind you might follow a broad area e.g. cardiology or specific e.g. cholesterol. As new content is added to TRIP we’d tweet it. It’s another way of people keeping up to date.

And still on the topic of Twitter, whom do you follow on it and what are some of your favorite Twitter tools?

I use tweetdeck which is wonderful. I follow around 250 people and these can be seen here .

You are considering adding a search wizard. Why? And how would it work?

This is another ‘wish list’ item. Currently, most searches on TRIP are single term, which is often problematic (search term vague = vague results). The search wizard would attempt to walk people through their clinical questions/scenario to unpick the important elements. We’d then use a modified version of the search to return a small number of results. Theoretically great, but in practice – we’ll see!

You also say, “Semantic analysis. We’re working hard to bring a semantic analysis function (a similar principle is used in PubMed’s related articles feature).” Would this resemble DeepDyve’s More Like This tool, which is pretty slick and could you do something like their Content Highlight tool

I discussed the related articles function earlier in the interview. I was not aware of deepdyve, but yes it’s a similar approach and that ‘more like this’ tool is relatively straightforward to set-up.

How do you see TRIP in a year? In five?

My primary aim is to make sure TRIP’s financial base is secure and we’re pretty confident. We get approached fairly regularly (1-2 times per year) from people wishing to invest. So we may take advantage of this one day. But aside from stability I want to keep developing TRIP to make it easy to use for clinicians and to ensure it serves their purposes. If they have a clinical question I want TRIP to answer it. Longer-term I’d like to see TRIP adopt a social aspect, some form of community and that can only make us stronger. Ultimately, I would like us to be even more widely used and still innovating.

What groups are not using it that you think really should be and why? Can you give a breakdown on usage numbers in terms of total traffic and by profession and specialty?

Who should use TRIP? Anyone who wants to locate robust research evidence to support their clinical practice! TRIP has been specifically designed for this function and it does it pretty well!

As mentioned earlier we get around 30-35,000 visits per day. Around 30% are from the UK and a similar number from the USA. The rest of the traffic comes from English-language countries (e.g. Australia, New Zealand and Canada, with India seeing a surge) also we get lots of traffic from Spain and South America. As for professions, 70% health professionals 30% non-health professionals. Of the 70% around 35% are physicians. As for specialties, we’ve no real idea. However, we are capturing this via My-TRIP so that’ll start giving us some decent statistics in the near future.

What aspect of TRIP are you most proud of and what has been your greatest challenge or frustration with it?

I’m most proud that we’ve managed to create a cutting-edge product that large numbers of people use to locate evidence to support their practice.
In other words we are helping to Turn Research Into Practice.

The challenge/frustration is one of funding. Being independent is great, but it has drawbacks. The NHS search (NHS Evidence) has £15 million funding a year while we operate on a lot less than 1% of that! We want to innovate, have lots of great ideas but not always the money to implement them.

Finally, who are your personal heroes in science, medicine, search, technology and in other field?

I’m not particularly into heroes as such. However, there are people I admire a great deal and have helped me directly or indirectly. To name a few I’d go for Noam Chomsky, Paul Glasziou (Centre for Evidence-Based Medicine, Oxford) and Muir Gray .

Thank you for your time.

It’s been a thought-provoking pleasure.