The Science of Quackometrics
So, how does the Quackometer work?
The quackometer counts words in web pages that quacks tend to use. The more quack words, the more quackery is suspected. That is Quackometrics.
The basic problem is that spotting the suspect words that many sites use, such as ‘vibrations’ or ‘energy’ is just not good enough as ‘good science’ sites are quite at liberty to use them. Even spotting these words in close conjunction with health terms, such as ‘healing’ or ‘nutrients’, is not quite good enough. My own background was research within in nuclear medicine group and the researchers had lots of legitimate reasons to mention ‘magnets’ and ‘health’ in (almost) the same breath.
So – the site uses an algorithm roughly like this:
- Keep a number of different dictionaries for use in tallying words in a web site
- Load the suspect web page and strip as much out as possible, HTML tags, scripts, punctuation etc.
- Count the number of words in each of the following dictionaries:
a) altmed terms: such as ‘homeopathic’, ‘herbal’, ‘naturopath’
b) pseudoscientific: clearly suspect terms that scientists rarely use such as ‘toxins’, ‘superfoods’.
c) domain specific words from biomed, physics or chemistry such as ‘energy’, ‘vibration’, ‘organic’.
d) skeptical words: words that no sincere homeopath would ever use, such as ‘placebo’, ‘flawed’, ‘crank’ or ‘prosecution’.
e) commerce terms that would indicate that something is for sale, such as ‘products’, ‘shipping’, or ‘p&p’.
f) Run a few other checks on pomo terms and religious terms, although not much is done with these. - Compare the ratio of frequency usage of these various types of terms and compare them to preset thresholds. If a threshold is exceeded then append the test’s associate sentence to the response. The tweaking I have been doing to the site has been adding words to dictionaries and varying the thresholds for matches.
This does not always work, Some quacks are very clever and avoid the obvious quack words. Nonetheless they still have completely hatstand ideas.
So, if anyone else has suggestions, then I would be very greatful. Just need to give up my real job to concentrate on this now.
Related posts:
- What is the Quackometer? The quackometer is an experiment to see if it is easy to spot quack web sites just from the language they use. The idea for this site came about after...
- What is Quackery? Definitions are hard. You could argue that one person’s quack is another’s health professional. I do not want to limit the definition to just those people who practice Complementary and...
- The Quest For The Perfect Quack! Well, I’ve had a bit of time to put some effort into the Quackometer, and now it has a few new features and improvements. 1. First – ‘Am I a...
- The Dilemma of Finding Reliable Health Advice Online A press release today from the Economic and Social Research Council talks about recent research on how the public evaluates the information on health websites. The concluding message is simple:...
Marvellous work.
Hearts and flowers to you. A healthy dose of skepticism is a wonderful thing.
The effort to identify and protect people from being fleeced is a noble one. On your Quackometrics page you begin to share your own perspective, that is, what kinds of medicine you believe to be valid. It would be more honest, however, if you shared your medical training and professional experience in treating patients in detail.
I’m sure readers would also like to know about your personal experience (or lack thereof) being treated by various forms of medicine. That is, be honest and reveal your own biases and basis before you set out to apply them to others. It’s easy to sit back and take potshots at other people; it’s much harder to roll up your sleeves and sign up for the laboratory of true scientific study and experimentation.
I really struggle about why so many commenters want to know my qualifications. This site and blog uses basic scientific principles and a smattering of reason to show why certain health claims are extremely suspect. One does not need to be a genius to realise that if, for example, no active ingredient is present in a homeopathic remedy, then it must be just a placebo. If you believe otherwise, then show the evidence, don’t ask for my biography.
To all the ‘anonymous’ posters out there. Please stick to the arguments and stop tying to resort to discussions on my qualifications, motives and so called biases. I will not be tempted into such futile discussions.
The description of your quackometer,
using the frequency of keywords
to detect whether a web page is quackery
or not, is reminiscent of the Bayesian
filtering techniques used in some antispam
software such as SpamAssassin (on Unix/Linux systems). These spam filters
using the frequencies of keywords in spam
messages compute (roughly speaking) the probability that a given email is spam
or non-spam and tag the message accordingly. They can be trained, given
a large enough sample of spam and non-spam message to discriminate better
between the two.
Another interesting application would be
a quack medicine generator, similar to the postmodern generator: you simply use the probabilites of keywords to generate
descriptions of new quack medicines.