Going back to a quote of John W. Tukey – who can be seen as the reviving power of statistical graphics, and thus ultimately of visualization in general – we can learn a bit about the motivation behind graphical data analysis

“… paradigm of exploratory data analysis
a) here is the data
b) what is it trying to tell us; in particular, which question does it want us to ask?
c) what seems to be going on?”

Although there seem to be the “data first” aspect in both the classical EDA approach and the modern data visualization, we can find a fine distinction regarding the motivation.

Here are two examples which are dominated by their flashy presentation, but fail to ask the relevant questions and can’t really tell us a story showing what seems to be going on, apart from what we (trivially) knew before.

This example is taken from the triposo website and shows locations of photos taken with smartphones and logged with the triposo trip advisory application. Whereas this is a cool visualization; what is it trying to tell us? From the comment on the website, we can see how badly the “story” behind the data fails: “This is probably the clearest example of all: Labor Day celebrations light up Europe and China in a big way. Who doesn’t want to take a picture of a nice 1st of May Parade? …” At least in the last 25 years, it was was hardtop find a single 1st of May Parade in Europe.

It gets even worse when the visualization actually shows things that are not in the data as in the next example from villevivante.

Nathan did post this example and finished with “It’s hard to say exactly what you’re seeing here because it does move so fast, and it probably means more if you live in or near Geneva, but speaking to the video itself, you have your highs and lows during the start and end of days.” It is not a particular insight that most of us travel into cities to go to work in the morning and move back out to our home at the end of the day. What is interesting though, is that according to the visualization, people in Geneva do not move along roads, but seem to enter the city like a swarm of bees …

To summarize, a good visualization should (at least) fulfill these requirements:

1. Be clear about what data was used (especially regarding generalization)
2. Make sure the visual abstraction does not lead to misinterpretations
3. Actually tell a story
4. Answer questions where we didn’t know the answer already

Nonetheless, there are two issues (which can be found in past posts on this blog), which summarize the most important facts on how the disaster was handled:

1. “Women and Children first!“

(each rectangle shows a combination of Class x Age x Sex) and
the proportion of survived passengers via highlighting)

2. Travelling 1st Class makes it easier to get into a life boat …

 (Launch sequence of life boats – woman highlighted)

The shocking result of this visualization is that the first 6 boats were exclusively used for 1st class passengers (although boat No. 1 was not filled at all), and it took another 4 boats until 3rd class passengers were considered to be saved. Starting with the 15th life boat, chaos spread, and the last three boats went afloat almost empty.

Looking at the two visualizations, shows clearly what fuels the stories (not only) in the Titanic movies. 2nd class males are among those with the lowest survival rate, indicating a very heroic attitude. The boat filling strategy shows a strong social bias, which can only partly be excused by the coincidence of passengers locations and boat locations.

(Here is the data on the passengers and boats used for the visualization)

The talk I chose to watch was by Nic Marks – as I tried to avoid yet another Hans Rosling talk.

After watching the talk I ended up at the happy planet index site, and found myself flipping through their report. On page 16, I found Figure 2, which kind of blew me away.

I don’t want to get into moral preachings here, but take your time and rethink your life and attitude and try to come up with some explanation and projection of what has changed over the last 45 years – I am curious to see your comments.

Sometimes even very simple statistics make you think hard …

On the JMP blog you find a post which uses the same data source I took for my first attempt to visualize the web of the EU debt. I found it hard to really make a point with this data. Looking at the JMP post tells me I wasn’t all too bad. Let’s walk through their visualizations:

1. The Map
   Certainly necessary for someone in the US – who knows where Portugal, Spain and Austria are; or was it Australia ..?
   But seriously, scaling the very irregular shapes of the countries outline has more problems (on the perceptual side) as benefits.
2. The Heat Map
   This graph degrades the information to a binary information of being creditor or not – be it 1€ or 1,000,000,000€. The conclusion that Germany and France are in trouble because they lend to many other countries is not convincing as it only reflects their economical power.
3. The Tree Map
   As tree maps are “only” a different representation of a tree, i.e., a hierarchy, why would you visualize a matrix with it? I can’t really  follow the chain reaction of defaults which is interpreted from this tree map. Looks a bit along the lines “if all you have is a hammer, every problem looks like a nail”.

One thing that is really not well thought about the graphs are the different color schemes between the graphs.

In the end, I think someone really needs to get “the right” data and show us “the right” visualizations such that we understand what’s going on – but hurry up, Greece might be broke by then …

At first sight there seems to be a contradiction between the apparently continuous color scale (see here for some thoughts on coropleth maps) and the map that does not seem to give any decent insight in the geographical distribution of population density. The answer is twofold.

1. The color scale is not continuous but has a break between green and blue (unless you invert the shades of blue) and blue and yellow. What we would expect – in less saturated colors – looks like this:

2. For a map showing a continuous quantity, we usually would not choose so many different saturated colors.

Let’s approach “The Good” as I still need to convince you that there might be a better version of the map. In a perfect world, coropleth maps look smooth and “continuous”. For the map of France we might want to look at the distance to the capitol Paris, as France is very centralistic. This map uses a monochromatic scale and shows “the perfect world” …

As this one is obviously too trivial, we want to look at the population density as in the above plot (2011 census data from wikipedia). Using a simple linear scale we would end up with this (useless) map, which uses a color scale that ranges from blue (small values) over white (median values) to red (large values):

Except for Paris and three other departments, all regions are unpopulated compared to the capitol. The extremely skewed distribution which is shown in the lower left, explains the dilemma.

Using the same “trick” as in the original wiki-map, i.e., cutting off all values above 150 we get a map that is easier to read, but now equalizes all information for areas above 150.

(Note, I used the histogram of log(population Density) for the legend)

The result is much better now, but there seem to be too many departments put into a single class.

From the data on the log-scale, we already see what would be most desirable, i.e., a distribution of colors, which is close to a normal distribution. Using a non-continuous transformation of the variable we display, we can map the color-shades to be normal, which ends up in the following map, which I would classify as “The Good”.

We now get a fairly good feeling of which regions are highly populated, which ones are close to the median (even with a distinction of being above or below average) and also clearly see the extremely unpopulated departments.

There is a lot more to say about the do’s and don’ts for drawing choropleth maps (which can be found here in Chapter 6). What is even more fun is to play around yourself! Here is the data (unzip and load France.txt with Mondrian) and here is the software – have fun!

(Thanks to Antony for providing the map!)

Given the trouble I got into after my post on the Japan earthquake, I probably should stay put when it comes to looking at data on hazardous events …

More seriously, as statistician (or data analyst in general) we often lack the expertise from the domain expert, who usually collected the data. Today, in a “data everywhere” world, we are in the fortunate position to easily access interesting data from various domains, but probably don’t know much about the background.

Thus I was happy to see the three posts

• Killer tornado perspective
• Tornado reports year in review
• Tornado days

on Jim’s blog. As Jim has a BS from SUNYA in Atmospheric Sciences, MS from FSU in Meteorology, and a PhD from ISU in Agricultural Meteorology, I am pretty sure he knows enough about tornados to reason beyond speculations.

You can find the data (5.5MB) here to play around yourself, which was compiled from this NOAA website. If you need a tool, you might be happy to use Mondrian.

PS: Jim agreed to write a guest post in the next few weeks, so we might learn a bit more on tornados here soon.

I had a good laugh when I saw Andrew’s reference to this barchart.

Maybe this is the right way to teach upper management the concept of uncertainty via confidence intervals, as the concept of mistrust is surely well known in these circles.

But to leave you with a bit of Christmas feeling, here is a great version of the ancient, latin Christmas hymn “Veni, Veni Emmanuel!”. Although I am not a particular bluegrass fan, I have to admit that this version is far closer to the original intention then what many well-meaning church choirs around will deliver.

-Enjoy!

Note: The video was recorded on Canon 5D Mark II, which gives you (given the right lenses) impressive depth of field effects – given you have the money to own a 5D MII.

There is also a short explanation to each country’s situation in relation to its GDP right of the graph, but that is full of interpretations and “insights” which hardly match with the figures in the graph.

After spinning around in the debt web, I keyed in all the data, and can now create the debt matrix:

I am not sure how much more I can seen now, but I see it now all at once at least. Surprisingly (or maybe not surprisingly) the two countries which would trouble me most, are not within the Euro-Zone and don’t seem to be part of any concerns: UK and US.

One last graph which looks at the influence of the highlighted countries, which already called for help and thus have quite some potential of defaulting on their debts:

The barchart shows creditors sorted according to the share of troubled debt – though I don’t feel enlightened enough to draw any immediate conclusion form this result … I guess the data does only show a small part of what’s going on and no matter how we visualize it, we are not really getting more insight into the crisis.

Maybe it takes another post with more / better data …

In the first place I thought the really bad thing (but maybe also amusing thing) of this “analysis” is the fact that some sample has been pushed through some multivariate statistical procedure and generated some output – many opportunities for failure and no idea about significance. While this kind of “analysis” (how did you find yourself in the two worlds?) might be somewhat frightening, the real frightening thing is the site, which generated the data.

Statistics can be of great help in many places, but we should not actively hand over our interests to the results of some data mining algorithm.

We find this nice illustration of German energy data at the GE visualization site:
This kind of visualization is quite common now and had its “initial public offering” with “The Baby Name Wizard” by Martin Wattenberg. The stacked display has some issues (which can make it to “a Bad”) and it takes a careful construction to make sure it is well readable (it actually “only” needs the right stacking order – if there is one). What struck me with above graphics was the fact, that none of the bands is actually aligned at some sort of straight base – typically the x-axis in a plot. As a consequence it is really hard to tell the story behind the data. Most frustrating, the most recent data is extremely jiggling which makes a judgement of the current trend almost impossible.

It took me a while to get the data out of the visualization, but you can actually download the whole visualization here. My first attempt to understand the data better was using simple time series which I created by “misusing” a parallel coordinate plot:
What we lose is the total, as the series are no longer stacked – though, it was quite hard to judge the total in the original visualization as well. The barchart is used as a reference and shows the most recent distribution. What can we learn from this graph:

1. Well, there was the oil crisis in 1973 – God knows what would have happened without the crisis stopping this ridicules greed for oil in the early 70s.
2. The second oil crisis in 1979 was actually having a real impact, as the decline in oil consumption lasted for four years and since then stayed on a lower level – quite contrary to the crisis in 1973.
3. Germany abandoned half of the brown coal sources shortly after the reunification.
4. Nuclear energy stalled in 2000 and is now on a (projected) decline.
5. Renewable energy sources are the only ones with a significant growth, but it still takes a long way to supersede oil and gas.
6. Coal is declining steadily.

You certainly can read off all the topics from the GE-visualization, but you probably would need to know these fact before, which is certainly the wrong way, as a visualization should generate insight and not visualize already existing knowledge.

PS: I tried to find a good stacking order, but after 30min. moving series up and down it looked like there is none.

PPS: There is a quite similar post here

But let’s start with an introductory example. The Titanic data is still the most convincing application of mosaic plots, though many of us saw this example over and over again – I will show other examples as well once we are done with it.

Above example starts with a simple bar chart of passengers by class at the top left, with all surviving passengers highlighted (I guess everybody is familiar with what happened to the Titanic …). The top right plot modifies the bar chart such that we can compare the highlighted proportions, i.e., the proportionality of width and height is interchanged, without changing the highlighting direction. We call this plot a spineplot.

With a spineplot, we are almost there for a 2-dim. mosaic plot, shown at the bottom of above graphic. Now we can derive the general building principle of a mosaic plot. We start with a blank rectangle and recursively split each tile according to the conditional distribution of the variable to add within that tile, e.g., we split the whole according to the distribution of class, and each class according to the second variable – in our case survived.

Leaving the survival information as highlighting, we can recursively split Class by Age and Gender and get the classical Titanic mosaic plot:

I guess it won’t take you long to find the “Women and Children first!” in the plot …

Now it is easy to see the fundamental difference to tree maps. Whereas in a tree map, we may split each node according to an individual criterion, the “tree” behind a mosaic plot is always fully balanced and the splits on a specific level are always according to the distribution of one fixed variable.

On the highest level, there are basically two general uses of mosaic plots.

1. Conditional Distributions
   Looking at a single response (like survival in the above example) or an interaction, conditioned on (or given a) set of variables (class x age x sex)
2. Structural properties of high-dim. categorical data
   Often we need to understand the general structure of a high-dim. categorical datasets in terms of finding empty or very small combinations, the dominating classes, or trends and patterns in the data.
   In this case we can make use of the numerous variations of mosaic plots (see, e.g., here for a Multiple Barchart), which mostly leave the strict area proportional constraint (which we need in 1.) and move to a matrix like layout (see Heike’s paper on more details, or try them out in Mondrian. See also Alex’s RMB-plots as latest contribution to this class of plots.)

Let me give you two more examples of mosaic plots. The first is using longitudinal categorical data on respiratory diseases.

For five points in time we see the different development of the disease depending on gender and kind of treatment, with highlighted cases marking patients with a “good” status. We see the highest discrimination between the treatments for t(2) for female patients and t(3) for male patients, and a decreasing effect for t(4) for both genders.

I will close with showing Simpson’s Paradox with the famous Berkeley admission data using mosaic plots:

The mosaic plot of gender with admitted students highlighted (left) shows clearly that the proportion of females is smaller than the one of males. If we split up by department (lower right plot) the share of admitted students is almost completely balanced for departments B-F and even higher for females in department A.

I leave it to the reader to find a neat verbal explanation of what is going one here (as this post is already way too long …), but so much can be said: it has to do with the proportion of females and males within the different departments.

With Mission Control, we now have all things in one place, and it is only the next swipe away to reach the desired functionality. I think this is a good example, that often we only lack the last missing link to get to the point where the UI functions fall into place – all of the single functions where released in previous OS releases before, but only now it is completely natural to use them all – and not just once in a while.

There is certainly the “one more thing” regarding UI changes in Lion: Natural Scrolling. Just search for the comments you find on the web – they reach from “Apple’s ‘natural scrolling’ feels horribly unnatural. Here’s why.” to “Wow, Everyone’s Complaining About “Natural Scrolling” In OS X Lion“. Well to be honest, it took me a few days to adopt as well, but once you are “over it”, it just works fine (even switching back and forth between the scroll wheel on my Win PC at work and my Mac at home). It is amazing how conservative people are regarding the way they use their computer – even if it is wrong. If we did it wrong for ten years, it has to stay that way … And there is no doubt about the fact, that there is really no physical metaphor behind the direction we used the scroll wheel so far – someone just programmed it this way and we used it.

Removing the scroll bars seems to be a comparatively small interference to the user’s expectation – still having enough potential to stir users up.

To sum up, with Lion we see how little progress we made with UI improvements in the last decades – but if we really leap forward, we feel the resistive force in the user base …