Books

• Jean Claude, Morphometry with R
  • Online through ETHZ
  • Chapter 3
  • Buy it
• John C. Russ, “The Image Processing Handbook”,(Boca Raton, CRC Press)
  • Available online within domain ethz.ch (or proxy.ethz.ch / public VPN)
• Hypothesis Testing Chapter

Papers / Sites

• Google/Stanford Statistics Intro
  • https://www.youtube.com/watch?v=YFC2KUmEebc
• MCB 140 P-value lecture at UC Berkeley (Audio)
  • https://itunes.apple.com/us/itunes-u/mcb-140-fall-2007-general/id461120088?mt=10
• Correlation and Causation (Video)
  • https://www.youtube.com/watch?v=YFC2KUmEebc
• Matlab Unit Testing Documentation
• Visualizing Genomic Data (General Visualization Techniques)

• NIMRod Parameter Studies

• M.E. Wolak, D.J. Fairbairn, Y.R. Paulsen (2012) Guidelines for Estimating Repeatability. Methods in Ecology and Evolution 3(1):129-137.

The course has covered imaging enough and there have been a few quantitative metrics, but “big” has not really entered.

What does big mean?

• Not just / even large
• it means being ready for big data
• volume, velocity, variety (3 V's)
• scalable, fast, easy to customize

So what is “big” imaging

• doing analyses in a disciplined manner
  • fixed steps
  • easy to regenerate results
  • no magic
• having everything automated
  • 100 samples is as easy as 1 sample
• being able to adapt and reuse analyses
  • one really well working script and modify parameters
  • different types of cells
  • different regions

There are two broad classes of data and scientific studies.

Observational

• Exploring large datasets looking for trends
• Population is random
• Not always hypothesis driven
• Rarely leads to causation

We examined 100 people and the ones with blue eyes were on average 10cm taller

In 100 cake samples, we found a 0.9 correlation between cooking time and bubble size

Controlled

• Most scientific studies fall into this category
• Specifics of the groups are controlled
• Can lead to causation

We examined 50 mice with gene XYZ off and 50 gene XYZ on and as the foot size increased by 10%

We increased the temperature and the number of pores in the metal increased by 10%

Coin Flip

1. Each flip gives us a small piece of information about the flipper and the coin
2. More flips provides more information
   • Random / Stochastic variations in coin and flipper cancel out
   • Systematic variations accumulate

Real Experiment

1. Each measurement tells us about our sample, out instrument, and our analysis
2. More measurements provide more information
   • Random / Stochastic variations in sample, instrument, and analysis cancel out
   • Normally the analysis has very little to no stochastic variation
   • Systematic variations accumulate

A very broad topic with plenty of sub-areas and deeper meanings. We mean two things by reproducibility

Analysis

The process of going from images to numbers is detailed in a clear manner that anyone, anywhere could follow and get the exact (within some tolerance) same numbers from your samples

• No platform dependence
• No proprietary or “in house” algorithms
• No manual clicking, tweaking, or copying
• One script to go from image to result

Measurement

Everything for analysis + taking a measurement several times (noise and exact alignment vary each time) does not change the statistics significantly

• No sensitivity to mounting or rotation
• No sensitivity to noise
• No dependence on exact illumination

The intraclass correlation coefficient basically looking at how similar objects within a group are compared to between groups

We have one biased coin and try to figure out how many flips we need for the ICC to tell the difference to the normal coin

With many thousands of flips we eventually see a very strong difference but unless it is very strongly biased ICC is a poor indicator for the differences

Once the reproducibility has been measured, it is possible to compare groups. The idea is to make a test to assess the likelihood that two groups are the same given the data

1. List assumptions
2. Establish a null hypothesis
   • Usually both groups are the same
3. Calculate the probability of the observations given the truth of the null hypothesis
   • Requires knowledge of probability distribution of the data
   • Modeling can be exceptionally complicated

Loaded Coin

We have 1 coin from a magic shop

• our assumptions are
  • we flip and observe flips of coins accurately and independently
  • the coin is invariant and always has the same expected value
• our null hypothesis is the coin is unbiased $E(\mathcal{X})=0.5$
• we can calculate the likelihood of a given observation given the number of flips (p-value)

How good is good enough?

Since we do not usually know our distribution very well or have enough samples to create a sufficient probability model

Student T Distribution

We assume the distribution of our stochastic variable is normal (Gaussian) and the t-distribution provides an estimate for the mean of the underlying distribution based on few observations.

• We estimate the likelihood of our observed values assuming they are coming from random observations of a normal process

Student T-Test

Incorporates this distribution and provides an easy method for assessing the likelihood that the two given set of observations are coming from the same underlying process (null hypothesis)

• Assume unbiased observations
• Assume normal distribution

Back to the magic coin, let's assume we are trying to publish a paper, we heard a p-value of < 0.05 (5%) was good enough. That means if we get 5 heads we are good!

Clearly this is not the case, otherwise we could keep flipping coins or ask all of our friends to flip until we got 5 heads and publish

The p-value is only meaningful when the experiment matches what we did.

• We didn't say the chance of getting 5 heads ever was < 5%
• We said if we have exactly 5 observations and all of them are heads the likelihood that a fair coin produced that result is <5%

Many methods to correct, most just involve scaling $p$. The likelihood of a sequence of 5 heads in a row if you perform 10 flips is 5x higher.

This is very bad news for us. We have the ability to quantify all sorts of interesting metrics

• cell distance to other cells
• cell oblateness
• cell distribution oblateness

So lets throw them all into a magical statistics algorithm and push the publish button

With our p value of less than 0.05 and a study with 10 samples in each group, how does increasing the number of variables affect our result

Using the simple correction factor (number of tests performed), we can make the significant findings constant again

So no harm done there we just add this correction factor right? Well what if we have exactly one variable with shift of 1.0 standard deviations from the other.

How does a standard image processing chain look?

• Green are the images we start with (measurements)
• Blue are processing steps
• Gray are use input parameters
• Pink are the outputs

• A mess, over 1080 combinations for just one sample (not even exploring a very large range of threshold values)
• To calculate this for even one sample can take days (weeks, years)
  • 512 x 512 x 512 foam sample $\rightarrow$ 12 weeks of processing time
  • 1024 x 1024 x 1024 femur bone $\rightarrow$ 1.9 years
• Not all samples are the same
• Once the analysis is run we have a ton of data
  • femur bone $\rightarrow$ 60 million shapes analyzed
• What do we even want?
• How do we judge the different results?

Given the complexity of the tree, we need to do some pruning

Qualitative Assessment

• Evaluating metrics using visual feedback
• Compare with expectations from other independent techniques or approach
• Are there artifacts which are included in the output?
• Do the shapes look correct?
• Are they distributed as expected?
• Is their orientation meaningful?

With a quantitative approach, we can calculate the specific shape or distribution metrics on the sample with each parameter and establish the relationship between parameter and metric.

Parameter Sweep

The way we do this is usually a parameter sweep which means taking one (or more) parameters and varying them between the reasonable bounds (judged qualitatively).

Sensitivity is defined in control system theory as the change in the value of an output against the change in the input. $$S = \frac{|\Delta \textrm{Metric}|}{|\Delta \textrm{Parameter}|}$$

Such a strict definition is not particularly useful for image processing since a threshold has a unit of intensity and a metric might be volume which has $m^3$ so the sensitivity becomes volume per intensity.

Practical Sensitivity

A more common approach is to estimate the variation in this parameter between images or within a single image (automatic threshold methods can be useful for this) and define the sensitivity based on this variation. It is also common to normalize it with the mean value so the result is a percentage.

$$S = \frac{max(\textrm{Metric})-min(\textrm{Metric})}{avg(\textrm{Metric})}$$

In this graph it is magnitude of the slope. The steeper the slope the more the metric changes given a small change in the parameter

Comparing Different Variables we see that the best (lowest) value for the count sensitivity is the highest for the volume and anisotropy.

Which metric is more important?

The first requirement for unit testing to work well is to have you tools divided up into small independent parts (functions)

• Each part can then be tested independently (unit testing)
  • If the tests are well done, units can be changed and tested independently
  • Makes upgrading or expanding tools easy
• The entire path can be tested (integration testing)
  • Catches mistakes in integration or glue

Ideally with realistic but simulated test data

• The utility of the testing is only as good as the tests you make

Example

• Given the following function function vxCnt=countVoxs(inImage)
• We can right the following tests
  • testEmpty2d

assert countVoxs(zeros(3,3)) == 0

• testEmpty3d

assert countVoxs(zeros(3,3,3)) == 0

• testDiag3d

assert countVoxs(eye(3)) == 3

There are too many graphs which say

• ‘my data is very complicated’
• ‘I know how to use __ toolbox in Matlab/R/Mathematica’

• Most programs by default make poor plots
  • Good visualizations takes time

1. What is my message?
2. Does the graphic communicate it clearly?
3. Is a graphic representation really necessary?
4. Does every line / color serve a purpose?
   • Pretend ink is very expensive

Simple Rules

1. Never use 3D graphics when it can be avoided (unless you want to be deliberately misleading), our visual system is not well suited for comparing heights of different
2. Pie charts can also be hard to interpret
3. Background color should almost always be white (not light gray)
4. Use color palettes adapted to human visual sensitivity

• Plots to “show the results” or “get a feeling” are usually not good

• Focus on a single, simple message
  • X is a little bit correlated with Y

• Too much data makes it very difficult to derive a clear message

• Filter and reduce information until it is extremely simple

• What is a grammar?
  • Set of rules for constructing and validating a sentence
  • Specifies the relationship and order between the words constituting the sentence
• How does this apply to graphics?
  • If we develop a consistent way of expressing graphics (sentences) in terms of elements (words) we can compose and decompose graphics easily

The most important modern work in graphical grammars is “The Grammar of Graphics” by Wilkinson, Anand, and Grossman (2005). This work built on earlier work by Bertin (1983) and proposed a grammar that can be used to describe and construct a wide range of statistical graphics.

H. Wickham. ggplot2: elegant graphics for data analysis. Springer New York, 2009.

Normally we think of plots in terms of some sort of data which is fed into a plot command that produces a picture

• In Excel you select a range and plot-type and click “Make”
• In Matlab you run plot(xdata,ydata,color/shape)

1. These produces entire graphics (sentences) or at least phrases in one go and thus abstract away from the idea of grammar.
2. If you spoke by finding entire sentences in a book it would be very ineffective, it is much better to build up word by word

Grammar

Separate the graph into its component parts

1. Data Mapping
   • $var1 \rightarrow x$, $var2 \rightarrow y$

1. Points
2. Axes / Coordinate System
3. Labels / Annotation

Construct graphics by focusing on each portion independently.

• I am not a statistician
• This is not a statistics course
• If you have questions or concerns, Both ETHZ and Uni Zurich offer free consultation with real statisticians
  • They are rarely bearers of good news

• Simulations are very helpful
• Try and understand the tests you are performing