Statistics 471/701 home page

General information

Class: MW3-4:30, F92
TA: Joshua Magarick
- email: statistics.assignments@gmail.com
- Joshua has a web page.
- Last year's TA Sathya, kept a web site with several useful files on it: code.
Office hours: Tuesday at 2pm (Huntsman 472). Or email me for an appointment.
Course work:
- submit all exercises to statistics.assignments@gmail.com
- Exercises (to introduce you to R. These are not that important--they are more for your benefit.)
- Homeworks (or more accurately, cases)
- Final project
- Note: You are the best students on campus. I have very high expecations on what you will learn this semester. This class is often listed as taking the most work of any class taken at wharton on student evaluations.
We will be using R (free) as our statistics package. I'll be using R in class. The book is on R. Statistics revolves around R.
Two useful books on R:
- Introductory Statistics with R by Peter Dalgaard, 2nd edition, ISBN 978-0-387-79053-4, Springer 2008 (paperback).
- Linear Models with R by Julian J. Faraway, ISBN 1-58488-425-8, Chapman & Hall/CRC Press 2005, (hardback)
But the web, and Joshua are your best resources!

Statistics 471/701

Schedule

Jan 11: Fitting functions with Taylor
- readings:
  - Richard Berk
  - Berndt chapter 2
  - Dalgaard: 2.4 and 6.1 (version 2 it is 1.6 and 5.1)
- Lecture notes: (.pdf, .html, and source=.Rnw)
Jan 13 @ 1-2 (F96): Introduction to R (by Joshua).
- bring your laptops!
- After seeing his introduction, you should be able to do the first practice R set. If you have questions about doing this, send us an email and we can add more information to the file.

Jan 16: No class (MLK)
Jan 17: (5pm) The first Practice R problems due
Jan 18: Doglegs
- Lecture notes: ( .pdf , .html and the source=.Rnw)
- read Richard Berk

Jan 22 (weekend!) Work on learning R. Do Josh's 2nd practice to confirm you are good with R. Turn it in on Wednesday. If you have R down, you should start on HW 1. At any rate, start reading Berndt chapter 2.
Jan 23: The residuals
- Notes: (.pdf, .html, .Rnw)
Jan 25: standard errors
- Read Dalgaard: Chapter 6 (6.2-6.4)
- Read Faraway: chapter 7
- Notes: (.pdf, .html, .Rnw)
Friday Jan 27 @ 1: 2nd introduction to R (in F85?)
- Check Josh's page for details. But he will be running a 2nd R introduction sometime friday afternoon.
- last years notes:
  - video
  - R code
  - events.csv (data used in the above example)

Jan 30: Bootstrap
- class notes: html, .Rnw
- Suggested readings for bootstraping data:
  - The Wiki article is very good
  - How to bootstrap in R.
  - A more theoretical description of bootstrap
  - Efron came up with the idea. See the book by Efron, Bradley. Tibshirani, Robert J. An introduction to bootstrap.
- To confirm you have understood the first 5 lectures, Start HW 2 (.html). No detailed write ups necessary--just R.
Jan 31: HW 1 due
Feb 1: BERNDT with bootstrap.

Feb 6: bootstrapping colinear data
Feb 8: Experiments and causation
- Read article by Jim Manzi.
- Discussion of Skinner vs. Piaget

Feb 13: Darwin day. (Applying natural experiments to the origins of sex.)
- handout
- wiki on the evolution of sex
- Widowbirds with long tails and the handicap principle
Feb 14: hw2 due (baseball data)
Feb 15: Obama and the SP 500 NEW!
- I fixed the block bootstrap! (See Rweave code)
- Here one picture I talked about in class: obama vs SP500.
- Justin Wolfers comment at Bloomberg (scroll to the bottom). (Hat tip to Joshua Lynn)
- If you want to play with this data here are the obama.csv and sp.csv data files.

Feb 20: Global warming day
Feb 22: Learning curves and Cobb-Douglas production functions
Feb 24: I'll be meeting with the Philomathean society for a tea. Come and join if you like!

Feb 27: Singularity and examples of learning curves
- questions from homework?
- Kurzweil view singularity (or on utube)
Feb 29: Stepwise regression: Introduction
- Wiki on model selection
- Wiki on Bonferroni
- A fairly readable research paper with Bob Stine
Mar 2: HW3: Fitting evolution data

SPRING BREAK!

Mar 12: You can mathematize anything!
Mar 14: Linguistics
- read the wiki page on Naive bayes.

Mar 19: Using R with words
- R output (R code)
- Due: Problems 1 and 2 from Berndt chapter 3.
- Pretty pictures: words in a cloud using R.
- Federalist data
Mar 21: Naive bayes implementation

Mar 26: PCA
- proposal for project due (.Rnw useful for cutting and pasting).
- This class will set up the concepts and the next class will show how to do it for linguistics and some R code.
- Bostrom and the not so happy future.
- Now this is big data
Mar 28: More PCA
- Source with latex
- Just R
- Reading: Stephen Stigler's article: " Statistics in 1933"

Apr 2: predicting the next word
- Zipf distribution (see NSF for some curious modern ideas on Zipf.)
- Ziff's law for the full wikipedia
- download the entire wiki.
CCA's
- handout
- Lit review due

Apr 9: summary of linguistics
- handout
- nouns vs verbs
- numbers
- R code for generating above plots
- data for above plots
Apr 11: Paranoid finance
- Linguistics assignment due
- book chapter
- an entire wiki on this topic

Apr 16: Slides on calibration
- Data analysis due
Apr 18: presentations

Apr 23: presentations
Apr 25: 3-5, G88 (reading days) presentations with allowed powerpoint

Apr 16: Data analysis due
- May 1st: Final write up due
Future Data sets
- If you can't access WRDS, here are the two files:
  - Fama deciles (with VW returns)
  - value weighted (aka SP500)
  - tbills
- Cleaning crews for practice reading into R
- Federalist papers
- Alice in wonderland: human readable, and word counts from google. Here are files that are easier to read into R: one, two.
- Google n-grams
Future readings
- Richard Berk
- Berndt chapter 2
- Dalgaard: 2.4 and 6.1 (in the previous edition: 1.6 and 5.1)
- Faraway: chapter 7
- Dalgaard: Chapter 6 (whole chapter)
- Dawkins: Handout
- Long term history of stocks / bonds.
Future practice exercises
I'll keep a page about the current R practice you should be doing.
- fire up R (first week's practice)
- make doglegs (second week's practice)
- residuals (3rd week's practice)
- hetroskadasticity (3rd week's practice)
- homework one help file.
- homework two help file.
- federalist help file
Sweave
Ask me about the Latex and R connection (Sweave)
- example source file
- What it looks like when processed
- Other references I've found useful:
  - powerdot,
  - My sample Sweave commands file: (.Rnw and .pdf)
  - An introduction
  - The Sweave manual itself
  - how to customize Sweave
Data sets used in 2009
- boston housing data (.txt, .jmp)
- New Hampshire Temp (.jmp) from Nature.
- Berndt data (for homework)
- Doing an infinite number of t-tests? (.jmp)
- Here is a simulation of an ideal hetroskadastic dataset.
- Online code for alpha investing (fast large variable selection).
- Loans (.csv)
Other Data sets of interest and some that will be used later
- World records for marathons by age Contrast with New england Journal of Medicine from 1980 (nejm198007173030304).
- palio global warming
- Display ft vs sales
- amazon data (from compscore)
- Population cohorts 1926 - 1979 (total .txt,.csv)
- DJIA (.txt, .csv)
- gold.jmp
- Homework 1: download Berndt data. (guidelines)
- Nerlov data (.jmp, .txt)
- learning model cars
- Berkshire Hathaway Inc (.JMP)
- info on hedge funds
- See the end of this page for other data sets of interest
- KOPCKE data for homework 4: (raw data) Note strange file format. You WILL need to edit it! If you read it into a text editor you will see that there are TWO different columns of number. Here are some comments on the file.
- Magic forecasting rule to generate excess returns
- IBM (.txt)
- value weighted.cvs
- Monthly T-bills, VW, inflation 1925 - 1995 (.jmp,.txt) suggestion: DJIA for homework. :-)
- Homework evaluations (.txt). Notice that no homework is statistically significantly worse than any other. Too bad!
- mink data (txt,txt, xls) documentation
- International Airline (.jmp)
- Unemployment (.csv)
- Live births in Pennsylvania 1915 - 1997 csv
- Fish (.jmp)
- Hurricane (Splus)
Last modified: Mon Apr 16 14:35:18 EDT 2012

General information

Statistics 471/701

Schedule

Future Data sets

Future readings

Future practice exercises

Sweave

Data sets used in 2009

Other Data sets of interest and some that will be used later