Self-Experiment Advantages

As a topic for self-experimentation, acne has many advantages:

• many highly-motivated subjects with ample free time

• objective visible effects where the data can even be easily recorded for blind ratings by third parties to reduce bias (ie. facial photographs)

• factorial experiments or blocking within-subject is possible for some acne interventions, which would considerably increase statistical power

  • given that acne is a long-term condition, one can establish no-intervention baselines and try multiple interventions simultaneously to speed things up; one can then test the final winner against a no-intervention baseline just in case

    • since you’re trying to find a cure for your personal acne (as acne has enormous individual-differences and so it seems likely that there is no one cure-all silver bullet), this makes sense to frame as a best-arm finding problem rather than a more conventional factorial which tries to estimate equally precisely all the interventions.

      One keeps a ‘winning’ intervention as the baseline, and then tries the next best intervention for a fixed period (eg. 1 month), and then updates, and pick the new winner + next-best for the next period.

  • one can self-control by randomizing half the face (divided vertically left/right); if there are concerns about systematic left-right differences, one can also randomized top/bottom, and so balance both simultaneously (by randomizing in a quartered pattern)

• in severe acne cases, changes would be easy to see within days/weeks, and large effects are plausible

• many already-known possible interventions, but none of which are particularly strongly supported (so high VoI); a good starting point would be the CureTogether crowdsourced rankings (see next section for specifics).

  • most of the interventions are safe & easy to implement: it’s not hard to use a face wash twice a day, apply some benzoyl peroxide cream, cut out all dairy products etc

• new commercial products allow for interesting hobbyist projects—what would microbiome sampling show about the microbial communities of teens with high acne vs low acne? About their microbial communities before the high acne group developed acne? After treatment with benzoyl peroxide or antibiotics or retinol? etc. With a large enough community dataset, interesting techniques could be applied for measuring heterogeneity in individual response to treatments (eg. GCTA-style variance components on acne response could be enlightening, as genes would be expected to be the source of much of the individual differences, acne itself being heritable)

• a launching pad into many areas of science & statistics—relevant topics include human & microbial genetics, methodological biases, regression to the mean, optimal experimental design, sequential testing, meta-analysis, time-series, deep learning… (“When making an axe handle with an axe, the model is near at hand.”)

  One could spin off lots of projects to increase rigor—instead of counting by hand, you could take smartphone photos and feed them into a pimple-counting CNN. With all the frameworks like PyTorch now, that’s well within the ability of a bright computer-inclined high schooler. (And then put it up as a web service so other teens can run their own self-experiments & score their pimple photographs, why not…)

History

Indeed, Seth Roberts, one of the early Quantified Self proponents, credits his interest in self-experiments to acne (Roberts2001/Roberts2010), when he found tetracycline didn’t help his acne but benzoyl peroxide did:

My interest in self-experimentation began when I read an article about teaching mathematics by Paul Halmos, a professor at Indiana University. Halmos emphasized that “the best way to learn is to do.” I was trying to learn how to do experiments; I took this advice to mean I should do as many as possible. I could do more experiments, I realized, if I not only did rat experiments but also did experiments with myself as the subject. So I started doing small self-experiments. Most of them were trivial and led nowhere (eg. experiments about juggling). At the time I had acne. My dermatologist had prescribed both pills (tetracycline, a wide-spectrum antibiotic) and a cream (active ingredient benzoyl peroxide). Simply for the sake of doing experiments, any experiments, I did simple tests to measure the effectiveness of these treatments. I believed the pills were powerful and the cream had little effect. To my great surprise, the tests showed the opposite: The cream was powerful and the pills had little effect. It was very useful information. Many years later, an article in the British Journal of Dermatology reported that antibiotic-resistant acne is common.

Acne came up several times on Roberts’s blog:

• Roberts has suggested that teens could run acne self-experiments as a group project: “Acne Clubs”

• A possible general cause: glycemic index/insulin signaling

• Anecdotes:

  • oolong/black tea allergy as acne trigger

  • pasteurized dairy trigger: 1, 2

  • sugar/carb triggers: 1, 2

  • soap trigger: 1; lack of soap trigger: 2

  • acne reduction due to milk thistle

CureTogether Acne Interventions

CureTogether (2008^–₄2012_12ya) was a social network/crowdsourced website for aggregating & ranking treatments of various problems, typically diseases, and had an acne page. They were bought by 23andMe in 2012, and operated for a while until the website seems to’ve vanished around mid-2016 & the data appears to no longer be available anywhere. The last Internet Archive snapshot of their acne page is 2016-04-01. (All inquiries I’ve made to 23andMe via Twitter & email have been ignored.)

CureTogether’s presentation of uncertainty & average ranking of the interventions is useless.

But they do provide the totals for each of the rating levels, so I decided to fix it by extracting & re-analyzing the ratings in a multi-level Bayesian ordinal regression with a weakly informative prior to get more useful posterior estimates of ratings.¹

## https://web.archive.org/web/20160401173234/http://curetogether.com/acne/treatments/
## "much worse / slightly worse / no effect / moderate improvement / major improvement" → 1--5
acne <- read.csv("https://gwern.net/doc/genetics/microbiome/acne/2016-04-01-curetogether-acne.csv",
                 header=TRUE, colClasses=c("integer", "integer", "factor"))
library(skimr)
skim(acne)
# Skim summary statistics
#  n obs: 439
#  n variables: 3
#
# ── Variable type:factor ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#      variable missing complete   n n_unique                     top_counts ordered
#  Intervention       0      439 439      113 Acn: 5, Agi: 5, Amo: 5, Ant: 5   FALSE
#
# ── Variable type:integer ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#  variable missing complete   n  mean    sd p0 p25 p50 p75 p100     hist
#    Effect       0      439 439  3.16  1.33  1   2   3   4    5 ▅▆▁▇▁▇▁▆
#         N       0      439 439 18.38 34.74  1   2   6  19  324 ▇▁▁▁▁▁▁▁
library(brms)
b <- brm(Effect | weights(N) ~ (1|Intervention), prior=c(prior(student_t(3, 0, 1), "sd")),
    family=cumulative(),
    iter=5000, chains=30, cores=30, data=acne)
b
# Group-Level Effects:
# ~Intervention (Number of levels: 113)
#               Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
# sd(Intercept)     0.64      0.06     0.53     0.76      16503 1.00
#
# Population-Level Effects:
#              Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
# Intercept[1]    −3.44      0.09    −3.62    −3.26      20848 1.00
# Intercept[2]    −2.17      0.08    −2.32    −2.02      17179 1.00
# Intercept[3]     0.49      0.07     0.35     0.64      15767 1.00
# Intercept[4]     2.75      0.08     2.58     2.91      19558 1.00
red <- as.data.frame(ranef(b)$Intervention)
round(red[order(red$Estimate.Intercept, decreasing=TRUE),], digits=2)

## the ordinal model forest plot doesn't make much intuitive sense, so redo as a normal-distribution
## for easier forest plotting (the rankings are more or less identical, perhaps a little less precise):
bg <- brm(Effect | weights(N) ~ (1|Intervention), prior=c(prior(student_t(3, 0, 1), "sd")),
    family=gaussian(),
    iter=5000, chains=30, cores=30, data=acne)
## Extract & sort estimates:
coefs <- as.data.frame(coef(bg))
coefs[order(coefs$Intervention.Estimate.Intercept, decreasing=TRUE),]
## Plot estimates:
library(brmstools)
forest(bg)