Quantitative Big Imaging

Big Data

• Google's Presentation on Distributed Computing
  • Slides
  • MapReduce Paper: Jeffrey Dean, et al. (n.d.). MapReduce: Simplified Data Processing on Large Clusters.
• Scalable Systems Course
• Tutorial in Hadoop
• Intro to Data Science @UCB

Cluster Computing

• Altintas, I. (2013). Workflow-driven programming paradigms for distributed analysis of biological big data. In 2013 IEEE 3rd International Conference on Computational Advances in Bio and medical Sciences (ICCABS) (pp. 1–1). IEEE. doi:10.1109/ICCABS.2013.6629243
• Condor High-throughput Computing
• Condor Setup at ITET
• Sun (now Oracle) Grid Engine or here

Databases

• Ollion, J., Cochennec, J., Loll, F., Escudé, C., & Boudier, T. (2013). TANGO: a generic tool for high-throughput 3D image analysis for studying nuclear organization. Bioinformatics (Oxford, England), 29(14), 1840–1. doi:10.1093/bioinformatics/btt276

Cloud Computing

• Amazon S3
• Sitaram, D., & Manjunath, G. (2012). Moving To The Cloud. null (Vol. null). Elsevier. doi:10.1016/B978-1-59749-725-1.00006-8
• Duan, P., Wang, W., Zhang, W., Gong, F., Zhang, P., & Rao, Y. (2013). Food Image Recognition Using Pervasive Cloud Computing. In 2013 IEEE International Conference on Green Computing and Communications and IEEE Internet of Things and IEEE Cyber, Physical and Social Computing (pp. 1631–1637). IEEE. doi:10.1109/GreenCom-iThings-CPSCom.2013.296

• Motivation
• Computer Science Principles
  • Parallelism
  • Distributed Computing
  • Imperative Programming
  • Declarative Programming
• Organization
  • Queue Systems / Cluster Computing
  • Parameterization
  • Databases

• Big Data
  • MapReduce
  • Spark
  • Streaming
• Cloud Computing
• Beyond / The future

There are three different types of problems that we will run into.

Really big data sets

• Several copies of the dataset need to be in memory for processing
• Computers with more 256GB are expensive and difficult to find
• Even they have 16 cores so still 16GB per CPU
• Drive speed / network file access becomes a limiting factor
• If it crashes you lose everything
  • or you have to manually write a bunch of mess check-pointing code

Many datasets

• For genome-scale studies 1000s of samples need to be analyzed identically
• Dynamic experiments can have hundreds of measurements
• Animal phenotyping can have many huge data-sets (1000s of 328GB datasets)
• Radiologist in Switzerland alone make 1 Petabyte of scans per year

Exploratory Studies

• Not sure what we are looking for
• Easy to develop new analyses
• Quick to test hypotheses

Zebra fish Full Animal Phenotyping

Full adult animal at cellular resolution 1000s of samples of full adult animals, imaged at 0.74 $\mu m$ resolution: Images 11500 x 2800 x 628 $\longrightarrow$ 20-40GVx / sample

• Identification of single cells (no down-sampling)
• Cell networks and connectivity
• Classification of cell type
• Registration with histology

Brain Project

Whole brain with cellular resolution 1 $cm^3$ scanned at 1 $\mu m$ resolution: Images $\longrightarrow$ 1000 GVx / sample

• Registration separate scans together
• Blood vessel structure and networks
• Registration with fMRI, histology

Normally when problems are approached they are solved for a single task as quickly as possible

• I need to filter my image with a median filter with a neighborhood of 5 x 5 and a square kernel
• then make a threshold of 10
• label the components
• then count how many voxels are in each component
• save it to a file

im_in=imread('test.jpg');
im_filter=medfilt2(im_in,[5,5]);
cl_img=bwlabel(im_filter>10);
cl_count=hist(cl_img,1:100);
dlmwrite(cl_count,'out.txt')

• What if you want to compare Gaussian and Median?
• What if you want to look at 3D instead of 2D images?
• What if you want to run the same analysis for a folder of images?

You have to rewrite everything, everytime

If you start with a bad approach, it is very difficult to fix, big data and reproducibility must be considered from the beginning

1. You have 10 friends who collectively know all the capital cities of the world.
   • To find the capital of a single country you just yell the country and wait for someone to respond (+++)
   • To find who knows the most countries, each, in turn, yells out how many countries they know and you select the highest (++)

1. Each friend has some money with them
   • To find the total amount of money you tell each person to tell you how much money they have and you add it together (+)
   • To find the median coin value, you ask each friend to tell you you all the coins they have and you make one master list and then find the median coin (-)

The largest issue with parallel / distributed tasks is the need to access the same resources at the same time

• memory / files
• pieces of information
• network resources

Dead-lock

Dining Philopher's Problem

• 6 philosophers at the table
• 6 forks
• Everyone needs two forks to eat
• Each philospher takes the fork on his left

Directly coordinating tasks on a computer.

• Languages like C, C++, Java, Matlab
• Exact orders are given (implicit time ordering)
• Data management is manually controlled
• Job and task scheduling is manual
• Potential to tweak and optimize performance

Making a soup

1. Buy vegetables at market
2. then Buy meat at butcher
3. then Chop carrots into pieces
4. then Chop potatos into pieces
5. then Heat water
6. then Wait until boiling then add chopped vegetables
7. then Wait 5 minutes and add meat

• Languages like SQL, Erlang, Haskell, Scala, Python, R can be declarative
• Goals are stated rather than specific details
• Data is automatically managed and copied
• Scheduling is automatic but not always efficient

Making a soup

• Buy vegetables at market $\rightarrow shop_{veggies}$
• Buy meat at butcher $\rightarrow shop_{meat}$
• Wait for $shop_{veggies}$: Chop carrots into pieces $\rightarrow chopped_{carrots}$
• Wait for $shop_{veggies}$: Chop potatos into pieces $\rightarrow chopped_{potatos}$
• Heat water
• Wait for $boiling water$,$chopped_{carrots}$,$chopped_{potatos}$: Add chopped vegetables
  • Wait 5 minutes and add meat

They look fairly similar, so what is the difference? The second is needlessly complicated for one person, but what if you have a team, how can several people make an imparitive soup faster (chopping vegetables together?)

Imperative soup

1. Buy {carrots, peas, tomatoes} at market
2. then Buy meat at butcher
3. then Chop carrots into pieces
4. then Chop potatos into pieces
5. then Heat water
6. then Wait until boiling then add chopped vegetables
7. then Wait 5 minutes and add meat

How can many people make a declarative soup faster? Give everyone a different task (not completely efficient since some tasks have to wait on others)

Declarative soup

• Buy {carrots, peas, tomatoes} at market $\rightarrow shop_{veggies}$
• Buy meat at butcher $\rightarrow shop_{meat}$
• Wait for $shop_{veggies}$: Chop carrots into pieces $\rightarrow chopped_{carrots}$
• Wait for $shop_{veggies}$: Chop potatos into pieces $\rightarrow chopped_{potatos}$
• Heat water
• Wait for $boiling_{water}$,$chopped_{carrots}$,$chopped_{potatos}$: Add chopped vegetables
  • Wait 5 minutes and add meat

Imperative

• optimize specific tasks (chopping vegetables, mixing) so that many people can do it faster
  • Matlab/Python do this with fast-fourier-transforms (automatically uses many cores to compute faster)
• make many soups at the same time (independent)
  • This leads us to cluster-based computing

Declarative

• run everything at once
• each core (computer) takes a task and runs it
• execution order does not matter
  • wait for portions to be available (dependency)

Lazy Evaluation

• do not run anything at all
• until something needs to be exported or saved
• run only the tasks that are needed for the final result
  • never buy tomatoes since they are not in the final soup

Queue processing systems (like Sun Grid Engine, Oracle Grid Engine, Apple XGrid, Condor, etc) are used to manage

• resources (computers, memory, storage)
• jobs (tasks to be run)
• users Based on a set of rules for how to share the resources to the users to run tasks.

Resources

• a collection of processors (CPU and GPU)
• memory, local storage
• access to bandwidth or special resource like a printer
• for a given period of time

Jobs

• specific task to run
• necessary (minimal/maximal) resources to run with
  • including execution time

Users

• accounts submitting jobs
• it can be undesirable for one user to dominate all of the resources all the time

A database is a collection of data stored in the format of tables: a number of columns and rows.

Animals

Here we have an table of the animals measured in an experiment and their weight

Cells

The cells is then an analysis looking at the cellular structures

Relational-databases can store relationships between different tables so the relationship between Animal in table Cells and id in table Animals can be preserved.

SQL (pronounced Sequel) stands for structured query language and is nearly universal for both searching (called querying) and adding (called inserting) data into databases. SQL is used in various forms from Firefox storing its preferences locally (using SQLite) to Facebook storing some of its user information (MySQL and Hive). So refering to the two tables we defined in the last entry, we can use SQL to get information about the tables independently of how they are stored (a single machine, a supercomputer, or in the cloud)

Basic queries

• Get the volume of all cells

SELECT Volume FROM Cells $$\rightarrow \begin{bmatrix} 1,2,0.95\end{bmatrix}$$

• Get the average volume of all cancer cells

SELECT AVG(Volume) FROM Cells WHERE Type = "Cancer" $$\rightarrow 0.975$$

We could have done these easily without SQL using Excel, Matlab or R

If we expand our SQL example to cellular networks with an additional table explicitly describing the links between cells

Table Representation

Now to calculate how many connections each cell has

SELECT id,COUNT(*) AS connection_count FROM Cells 
  INNER JOIN Network ON (id=id1 OR id=id2)

$$\rightarrow \begin{bmatrix} (1 & 3) \\ (2 & 1) \\ (3 & 2)\end{bmatrix}$$

Basic networks can be entered and queries using SQL but relatively simple sounding requests can get complicated very quickly

How many cells are within two connections of each cell

SELECT id,COUNT(*) AS connection_count FROM Cells as CellsA
  INNER JOIN Network as NetA ON Where (id=NetA.id1)
  INNER JOIN Network as NetB ON Where (NetA.id2=NetB.id1)

This is still readable but becomes very cumbersome quickly and difficult to manage

NoSQL (Not Only SQL)

A new generation of database software which extends the functionality of SQL to allow for more scalability (MongoDB) or specificity for problems like networks or graphs called generally Graph Databases

some people claim to have had the idea before, Google is certainly the first to do it at scale

Several engineers at Google recognized common elements in many of the tasks being performed. They then proceeded to divide all tasks into two classes Map and Reduce

Map

Map is where a function is applied to every element in the list and the function depends only on exactly that element $$\vec{L} = \begin{bmatrix} 1,2,3,4,5 \end{bmatrix}$$ $$f(x) = x^2$$ $$map(f \rightarrow \vec{L}) = \begin{bmatrix} 1,4,9,16,25 \end{bmatrix}$$

Reduce

Reduce is more complicated and involves aggregating a number of different elements and summarizing them. For example the $\Sigma$ function can be written as a reduce function $$\vec{L} = \begin{bmatrix} 1,2,3,4,5 \end{bmatrix}$$ $$g(a,b) = a+b$$ Reduce then applies the function to the first two elements, and then to the result of the first two with the third and so on until all the elements are done $$reduce(f \rightarrow \vec{L}) = g(g(g(g(1,2),3),4),5)$$

They designed a framework for handling distributing and running these types of jobs on clusters. So for each job a dataset ($\vec{L}$), Map-task ($f$), a grouping, and Reduce-task ($g$) are specified

• Partition input data ($\vec{L}$) into chunks across all machines in the cluster $$\downarrow$$
• Apply Map ($f$) to each element $$\downarrow$$
• Shuffle and Repartition or Group Data $$\downarrow$$
• Apply Reduce ($g$) to each group $$\downarrow$$
• Collect all of the results and write to disk

All of the steps in between can be written once in a robust, safe manner and then used for every task which can be described using this MapReduce paradigm. These tasks $\langle \vec{L}, f(x), g(a,b) \rangle$ is refered to as a job.

Using MapReduce on a folder full of text-documents.

• $$\vec{L} = \begin{bmatrix} "\textrm{Info}\cdots", "\textrm{Expenses}\cdots",\cdots \end{bmatrix}$$
• Map is then a function $f$ which takes in a long string and returns a list of all of the words (text seperated by spaces) as key-value pairs with the value being the number of times that word appeared
• f(x) = [(word,1) for word in x.split(" ")]
• Grouping is then performed by keys (group all words together)
• Reduce adds up the values for each word

L = ["cat dog car",
  "dog car dog"]

$$\downarrow \textbf{ Map } : f(x)$$

[("cat",1),("dog",1),("car",1),("dog",1),("car",1),("dog",1)]

$$\downarrow \textrm{ Shuffle / Group}$$

"cat": (1)
"dog": (1,1,1)
"car": (1,1)

$$\downarrow \textbf{ Reduce } : g(a,b)$$

[("cat",1),("dog",3),("car",2)]

Technical Specifications

• Developed by the Algorithms, Machines, and People Lab at UC Berkeley in 2012
• General tool for all Directed Acyclical Graph (DAG) workflows
• Course-grained processing $\rightarrow$ simple operations applied to entire sets
  • Map, reduce, join, group by, fold, foreach, filter,…
• In-memory caching

Zaharia, M., et. al (2012). Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

Practical Specification

• Distributed, parallel computing without logistics, libraries, or compiling
• Declarative rather than imperative
  • Apply operation $f$ to each image / block
  • NOT tell computer 3 to wait for an image from computer 2 to and perform operation $f$ and send it to computer 1
  • Even scheduling is handled automatically
• Results can be stored in memory, on disk, redundant or not

• Local resources are expensive and underutilized
• Management and updates are expensive and require dedicated IT staff

Cloud Resources

• Automatically setup
• Unlimited potential capacity and storage
• Cluster management already setup
• Common tools with many people using the same

Needs

• Scale up to 1000GVx samples (eventually)
• Analyze standard measurements 14GVx regularly in a day
• Analyze as fast as we usually measure 15 minutes / scan

Would be nice

• Analyze scans as fast as we measure now (1 scan / minute)
• Analyze scans as fast as we could measure with Gigafrost (10 scans / minute)

• Exploring phenotypes of 55 million cells.
• Our current approach has been either
• Group and divide then average the results.
  • Average Cell Volume, etc.
  • Average density or connectivity
• Detailed analysis of individual samples
  • K-means clustering for different regions in bone / layers
  • Principal component analysis for phenotypes

If we do that, we miss a lot!

The results in the table show the within and between sample variation for selected phenotypes in the first two columns and the ratio of the within and between sample numbers