Apple Health Data How to Export Analyze Visualize Guide

apple_health_kit_steps_by_day_of_week_r_graph

Introduction

Did you know that the Apple Health App is recording your steps, mileage, and flights climbed data? This post will show you how to export, analyze and visualize your Apple Health data using both Excel and R. But first let’s establish that you may be sitting on a mountain of personal fitness data on your iPhone without even knowing it.

Send me your Apple Health export.xml file and I’ll analyze your fitness data for you.

Contents
Introduction
Export Apple Health App Data
Transform Apple Health XML to CSV
Analyze and Visualize Apple Health Data in Excel
Analyze and Visualize Apple Health Data in R

On iPhones 5s and newer and iOS 8 and newer, the health data is automatically collected as long as you have your iPhone with you when you are walking, running or hiking. Senors on the iPhone including the gyroscope, accelerometer, GPS, and barometer are used to measure steps, miles and flights climbed. To access the Apple Health data find and tap the heart health icon in your applications shown below. Your quantified self data is already being captured and you may not even know it.

AppleHealth_icon_

By default your Health dashboard should launch. If not tap the Dashboard icon on the bottom navigation. The dashboard will look like the screen shot shown below.

apple_health_dashboard

If you tap on any of the graphs twice, the first tap will bring you to a screen with a graph of just the single metric i.e. steps, and the second tap will launch a detailed table view of the data by day. This gives a nice quick view of your health data, but it makes it difficult to answer questions such as:

How do my steps this month compare to the same month to last year?
Am I more or less active on the weekdays or weekends?

The boxplot below shows my steps data by day of the week by year. This visualization was created using R and is just one example of what you will learn to create in this post.

apple_health_kit_steps_by_day_of_week_r_graph

This post will show you how to answer these questions and get started exporting, analyzing and visualizing your Apple Health steps, walking and running distance and flights climbed data.

How to Export Apple Health App Data

1) Launch the Apple Health App on your iPhone

2) Tap the Health Data icon in the bottom navigation. This will launch a list of all your Apple Heath data. In the List view, tap the “All” item which is the first item in the list.

apple_health_export_1

3) Tap the send arrow icon in the top right. This will launch an alert that says “Exporting Health Data Preparing…” The export preparation took 4 minutes for my health data, so be patient.

apple_health_export_2

4) Once the data is ready to send you’ll see an overlay where you can select how to send the health data export. I chose to send the file via email. The file name is export.zip.

apple_health_export_3apple_health_export_4

5) Download the export.zip file (my file was 4.3 MB). Then unzip the file. After you unzip you should see two files: export.xml (my file was 100 MB) and export_cda.xml. For this analysis we will just use the export.xml. Extract and save your export.xml file to a new folder. I called my folder apple_health_data and the folder is on the desktop.
Open the export.xml file in a text editor and at the very top you’ll see header info. When you scroll down in the file you’ll see your health data with records for step count, flights climbed and distances walking/running. See the below examples of single record rows for each of the three different standard iPhone Apple Health data points. If you scroll all the way to the right below you’ll see the creationDate, startDate, endDate, and value (steps, distance, flights).

Now that you have a full export of all your Apple Kit Health data you will need to transform the XML data into a CSV file for analysis.

How to Transform your Apple Health Data from XML to CSV

To transform your Apple Health export.xml data to a useable CSV format we will use a Python script. The Python script will take the export.xml file and create three CSV files with your Apple Health data: StepCount.csv, DistanceWalkingRunning.csv, FlightsClimbed.csv.

If Python isn’t your jam you can also use this web based Apple Health XML to CSV Converter.

If you are going to use R to analyze and visualize your Apple Health data, you can skip this Python XML to CSV data transformation step. The R script will transform and load your Apple Health XML data into an R data frame for analysis.

1) Copy and save the Python script below as applehealthdata.py in the apple_health_data folder we created previously. The Python script needs to be saved in the same folder as the export.xml file. Thanks to the Test-Driven data analysis blog for creating this Python Script.

2) You will have to run the applehealthdata.py Python script via the command line. To open the command line on Windows click Start > Run > cmd or search for and run cmd. To run the script you need to enter: #1 path to Python #2 path to applehealth.py script #3 path to the export.xml file then press enter. You can see the path to each of these three items on my computer below.

Let the script run. If everything is successful you will see summary statistics in the command line showing the count for the fields and records included in your CSV files. You’ll also see the path for your three Apple Health data CSV files. The StepCount.csv, DistanceWalkingRunning.csv, and FlightsClimbed.csv files are written to the apple_health_data folder (directory).

apple_health_python_commandline_script

Below is a screen shot of the StepCount.csv file opened in Excel.

apple_health_excel_step_data

StepCount.csv Fields and Sample Data

  • sourceName – Ryan Praskievicz iPhone
  • sourceVersion – 9.3.2
  • device – <<HKDevice: 0x161169930>, name:iPhone, manufacturer:Apple, model:iPhone, hardware:iPhone8,1, software:9.3.2>
  • type – StepCount
  • unit – count
  • creationDate – 6/1/2016  7:22:22 PM
  • startDate – 6/1/2016  7:06:51 PM
  • endDate – 6/1/2016  7:07:52 PM
  • value – 125

How to Analyze and Visualize your Apple Health Data in Excel

The Apple Health Dashboard view is great for getting a quick snap shot of daily, weekly, monthly and yearly data on Steps, Distance, and Flights climbed. To explore the underlying data for each graph you can tap the graph and see a table of daily totals for your health metrics. One drawback of viewing the data in the health app is that if you choose a monthly or yearly graph view, when you drill into the table view, the underlying data is still shown with daily granularity (not monthly or yearly). This makes it difficult to do comparisons month over month. You can eyeball the health kit graph and see that it looks like your total number of steps this month are greater than your total steps last month, but it is hard to know any more specific details. Digging deep gives you better insight into your quantified self data.

apple_health_dashboard

1) As a first step let’s use Excel to recreate the Apple Health dashboard and enhance the visualization and data table to give us a more detailed view of our steps. Open your StepCount.csv file and go to the Insert Tab and click PivotTable. Excel should be able to preselect your steps data set including the headers with the PivotTable Wizard.

apple_health_insert_pivot_table

2) Now that we’ve defined the source data for our PivotTable let’s select the fields we are interested in for analysis. From the PivotTable Field List choose endDate and value. The endDate should be in the PivotTable Row Labels and the Sum of value should be in the PivotTable Values. You should now see a PivotTable with with date time stamps in the first column and steps in the second column.

3) We are interested in seeing the data by month, so we need to aggregate the multiple rows per day of step measurements into monthly step data for each year. Right click on the PivotTable data and click on Group as shown in the screen shot below.

apple_health_pivot_table_groupbythe

When you click on the Group… link shown above this will open a Grouping dialog box. For your Grouping choose Months and Years and click Ok as shown in the screen shot below.

apple_health_pivot_table_groupby_months_years

You should now see a PivotTable showing your steps data by month for each year. The first full month of steps data for me was October 2015.

apple_health_steps_by_year_by_month

4) Let’s reorganize the PivotTable to make it easier to compare the Steps data by month year over year. In the PivotTable Field List click on Years under the Row Labels and drag Years to Column Labels. See the screen shot below showing the PivotTable data now organized with Months in the Rows and Years in the Columns.

apple_health_steps_by_year_by_month_compare

5) Now let’s add a graph showing steps data by month that let’s you easily compare year over years data. Click anywhere on your PivotTable to activate the PivotTable Tools context menu, then click on PivotChart to add a graph. I chose the default bar graph.

apple_health_steps_by_year_by_month_insert_pivot_chart

When you insert the PivotChart you will see a graph similar to the graph below (I used the blue scale chart design and removed the grid lines to clean the graph up). This graph and the PivotTable now make it easy to compare my Apple Health steps data month over month by year.

apple_health_steps_by_year_by_month_insert_graph

6) To view all of your step data by day Group the PivotTable data by Day and Year. Then to see a graph with all of your steps data add the PivotChart and make sure that Years and endDate are both in the Axis Field. All of my daily Apple Health Step data is shown in the graph below.

apple_health_steps_by_year_by_day_graph

How to Analyze and Visualize your Apple Health Data using R

Using R to analyze and visualize your Apple Health data gives a repeatable programmatic option compared to Excel. No previous programming experience or R knowledge is required for this tutorial, just like the other tutorials on my blog. I walk you through an R script that you can use to analyze your Apple Health data.

1) First you need to download and install R.

2) Next you have to Install R Studio. R Studio is an integrated development environment (IDE) which is the interface you’ll write, run and interact with your code. R Studio has a set of tools to make it easier to work with R.

3) Launch R Studio and install the R packages we will use in this tutorial: dplyr , ggplot2 ,  lubridate, and XML. In the Console module in R Studio (the bottom left pane) run the code below. Copy and paste or type the code shown below and press enter. This will install the packages via an online package repository called the Comprehensive R Archive Network or CRAN.

Once these packages have installed successfully you’ll see a message in the console that says “package successfully installed…”

4) Download and Run the Apple Health R Analysis Script

Save or copy the script below to your computer and open it in RStudio. In RStudio in the top left pane you’ll see the code, click on the Source button in the top right of the code pane. This will run the entire script. You’ll see tables and information printed to the R Console in the bottom left pane and you’ll see a heatmap in the bottom right Plots pane. To page through the visualization click on the forward and backward arrows above the Plots. This R script will produce 4 plots.

In the rest of the post I’ll walk you through each section of the code. To run a section of the code instead of the entire script, highlight the lines of code and click the Run button next to the Source button in the top right of the code pane.

5) Load the export.xml data into a data frame in R make sure to specify the path to the file on your computer on line 7.

Line 18 of the code converts the endDate 2016-06-02 12:27:46 -0400 into a type of POSIXct format variable that accounts for the time zone. I am in New York City so I set  tz="America/New_York" Check out the examples in the lubridate docs to see what you should set for your time zone.

In lines 22-26 we add month, year, date, day of week, and hour of day to our df data frame as new columns. This is important because the endDate for Apple Health measurements (Steps, Miles, Flights Climbed) can be as little as seconds apart so this granular data needs to be aggregated into a more manageable time range for analysis.

6) Graph Steps by Month by Year in R Bar Graph

How do my steps this month compare to the same month last year? We need to aggregate our raw steps data by year and month to answer this question. The dplyr package group_by()  function groups the steps data by year and month, two of the new columns we added to our df data frame in step 5 above. The summarize()  function is one of the 5 “verbs” of the dplyr package. It is used with group_by() to summarize (verb), in this case sum our grouped steps data on line 33. The table below the code snippet which shows the aggregate steps data by year by month. This is printed to the console by the code on line 35.

Next let’s add a bar graph to visualize the steps data by year by month. We are going to use the ggplot2 package to create the bar graph. Define the plot aesthetics aes on line 37 setting the x axis data equal to the month and the y axis equal to the steps. The fill aesthetic is set equal to year so each year on the bar graph is filled with a different color. Next define the type of graph geom_bar() on line 38 and set position='dodge' to tell the yearly bars for each month to dodge (be placed next to each other) and stat='identity' is set to map the heights of the bars in the graph to the values of steps (the y axis data).

Rather than using the default color scale for the graph on line 40 I set  scale_fill_brewer() to get a cleaner color scale. This sets a blue color scale with darker and lighter blues representing each year. Alternatively to use a grey scale you can set line 40 to  scale_fill_grey()

To remove the grid lines to clean the graph up even more, on line 42  theme(panel.grid.major = element_blank()) sets the grid line panel element to blank. Finally, to make sure Steps are not shown as a number formatted in scientific notation on line 39 specify scale_y_continuous(labels = scales::comma)

Quick aside on pipes %>% and dplyr

Using pipes  %>% and dplyr lets you chain functions together. Chaining functions allows you to avoid storing data in temporary data frames and simplifies your code. Pipes in the code are read as “then”. So reading through the code starring on line 30 below and putting it into words: we are telling R to take the df data frame then group the df data frame by year and month then summarize this grouped data by the sum of the steps then print the results then graph this monthly steps data by year by month.

Steps Data by Year by Month R Bar Graph

Some quick analysis of my monthly steps data. I had fewer steps per month the first 5 months of 2016 as compared to 2015 in all months except February. I have been biking more than walking and running so far in 2016 so this makes sense. Also, July 2015 I had over 500,000 steps. I was on a trip to Europe where I walked and hiked a bunch so that explains the spike.

apple_health_kit_steps_by_year_by_month__r_bargraph

Steps Data by Year by Month R Data Table

7) Graph Steps by Month by Year in R Boxplot

A boxplot is another more detailed option to visualize monthly steps data. The boxplot shows the variablity of the daily steps data for each month rather than just the totals by month shown in the bar graph. This allows us to dig deeper into our quantified self data. The code below should look similar to the steps data grouping and bar graph code above. The first difference in the boxplot code is that we now groupby()  date as well as month and year on line 47 so we have our granular date level data underlying our boxplot. The second difference in the boxplot code is geom_boxplot(aes(fill=(year))) on line 52 which generates the boxplot instead of the bar graph.

Also, included starting on line 57 is code to generate the summary statistics for steps by month for 2015. This gives you mean, standard deviation, median, maximum, minimum, 1st Quartile, 3rd Quartile steps summary statistics for each month in a table form. The table gives you the chance to dig deeper on trends that you identify in the boxplot. The data table is output to the R Console and I’ve included my data below.

Steps Data by Year by Month R Boxplot

Extending the analysis of my monthly steps by year data from the bar graph, in the boxplot I can see the max steps data point of over 30k steps in July 2015. The outliers in the boxplot are denoted with dots. In the summary statistics table below the graph, the actual max number of steps for July 2015 was 35,582 steps. Also, in May 2015 my median 12,405 steps for the month was greater than the 3rd quartile of steps for May 2016 11,441 steps. This makes sense because I didn’t have my phone with me all day everyday in May 2016 due to a camping trip.

apple_health_kit_steps_by_month_by_year_r_boxplot

Steps Summary Statistics Data by Month for 2015 R Data Table

8) Graph Steps by Day of Week by Year in R Boxplot

This is another boxplot which visualizes the day of week year over year. This is very similar to the previous boxplot with a few minor code changes. To create the boxplot by day of week instead of month use dayofweek in the  groupby() on line 72 and in the ggplot() on line 76 we use dayofweek on our x axis instead of month.

Just like the previous boxplot there is code for steps summary statistics to be output to a table. The summary statistic table code starts on line 82. On line 93  arrange(desc(median)) sorts the median steps by day of week from greatest to least. This makes it easier to see what days of week you are most and least active.

Steps Data by Year by Day of Week R Boxplot

One thing that jumps out to me is that for 2015 Thursday 13,187 median steps and Saturday 12,724 median steps are the top two days of the week for 2015. Saturday in 2015 also has the greatest standard deviation of any day. This makes sense because there were Saturday’s in 2015 when I went on long hikes and some Saturdays when I relaxed and took days off from exercise. One other obeservation is the similarity year of year in steps on Sunday, Monday and Tuesday. The boxplots are all relatively similar for those 3 days with the major difference on Tuesday’s in 2016 with the median steps much lower than Tuesdays in 2014 and 2015.

apple_health_kit_steps_by_day_of_week_r_graph

Steps Summary Statistics Data by Day of Week for 2015 R Data Table

Bonus

I’ve added a heatmap visualization to the R Script starting on line 95. This heatmap shows your steps data visualized by day of week and hour of day. It is a nice way to see patterns in your daily activity.

Summary

I hope this post gave you the necessary tools to export, analyze and visualize your Apple Health Steps data. You should be able to apply the Excel analysis and the R sample code and analysis to the walking and running distance and the flights climbed CSV files to analyze those two data sets as well. To run the R script on miles walked/run data swap HKQuantityTypeIdentifierStepCount for HKQuantityTypeIdentifierDistanceWalkingRunning and for flights climbed data use HKQuantityTypeIdentifierFlightsClimbed. Hopefully this tutorial also helps you to understand the value of using R and other programmatic data analysis tools as compared to Excel. And my final wish is that by analyzing your Apple Health data, you take action. Go ahead and set goals for yourself, and use the quantified self data as motivation to be active.

Please share any questions, comments or ideas for deeper analysis of the Apple Health Kit data with me in the comments below, on Twitter @ryanpraski or email ryan@ryanpraski.com.

  • Jed Fan

    This is great thanks for taking your time to write this Ryan

  • martintengelin

    Hello Ryan! Love this post and R script, thanks a lot for sharing! I do however have a problem/question that I hope you can help with. The sleep data doesn’t seem to be captured in the resulting dataframe (the “value” column is all na:s), although I can see it in Apple Health. Have you also experienced this issue, and/or do you know why? Thanks!

    • Hi Martin,

      Can you give me a sample of how your sleep data looks in the Apple Health export.xml file. If you don’t want to share here you can email it to me: ryanpraski@gmail.com

      -Ryan

  • Cindy Liu

    This is awesome! Thanks for the article Ryan. I’m so happy to be able to see my steps over a longer period of time now.

  • Tom Tao

    Ryan – This is a great post. I love my Apple Watch and plan to play around the data a bit. Your post is definitely a primer read.

    One question for curiousity – Why don’t you use Python for analysis and drawing chart? To the best of my knowledge, Python and its library provide some neat visualization option as well.

    Tom
    http://www.bubbletao.com