As a general rule, this course is graded on a “Check Plus” / “Check” / “Check Minus” / “Fail” grading scale. These roughly correspond to a 4.0, 3.5, 3.0, and 1.0, respectively.
Below, I roughly describe criteria for what sophisticated, compentent, and not competent are for different aspects of the assignments in this course.
Sophisticated statistics use appropriate statistics for the argument being made. The statistics used help make patterns in the data clear to the reader. Appropriate units are reported for all statistics, and are interpreted in terms of the research questions and real-world significance. Often, multiple different statistics are used to show how different analyses show different aspects of the data, and these are explained clearly. Assumptions are understoond, and arguments are made about how the data either meets the assumptions, or how the assumption violation might affect (or not affect) the results. Uses simpler statistics when appropriate to make arguments clearer, but backs them up with more sophisticated statistics as necessary.
Competent statistics uses an appropriate statistic for the type of data (e.g. t-test for two-group continuous data). It correctly and validly presents the results of the statistic and interprets it in light of the reseearch question(s). The data and use meets appropriate assumptions for the statistic. Examines the effect size appropriately rather than just focusing on statistical significance.
Not yet competent statistics do not match statistic choice to the data. These statistics throw away or ignore important parts of the data (for example, binning a continuous variable to do a chi-square test) without having a good reason. Choosing statistics based on their ability to provide “stars” or p-values. Following statistical instructions because some book or guide tells you do without understanding why.
Sophisticated use of hypothesis tests is as a secondary argument. The main argument in the paper should be about the real-world effect sizes, and the hypothesis tests are either not used, or only used to discuss the amount of noise in the data. Units are always taken into account, and p-values are never compared to each other nor discussed as “very significant” or “marginally significant”.
Competent use of hypothesis tests combines hypothesis tests with effect sizes to make arguments. While a hypothesis test might help identify whether the analyst should pay attention to a result, the analyst also examines the effect size to identify the meaning of the difference. Units are always taken into account, and p-values are never compared to each other nor discussed as “very significant” or “marginally significant”.
Not yet competent use of hypothesis tests only focuses on the statistical significance (or “stars”), and ignores the actual effect size. The absolute value of the p-value is given outsized meaning (such as “very significant” or “marginally significant”), and p-values may be compared with one another (or a “significant” finding compared with a “non-significant” finding).
Sophisticated modeling uses appropriate transformations and explanations to help the reader understand the results. Categorical variables have their baseline level carefully chosen for clear analyses, and it is always recognized that coefficients are comparisons against that baseline. Binary variables are named appropriately for easy interpretation. Continuous variables have clear and understandable units, and are interpreted correctly in real-world units. All model parameters are reported, and the model is parsimonious. There is a discussion of control variables, and what it means to control for those variables.
Competent modeling reports on complete regression models, and includes coefficients for all model parameters. Categorical variables are correctly interpreted as comparisons against a baseline. Continuous variables are reported with units. Interactions are correctly interpreted, and main effects are interpreted correctly relative to other predictors.
Not yet competent modeling talks about how variables are “significant” rather than the estimates. Categorical variables are described as if they are significant themselves, without reference to a baseline level. Continuous variables are discussed without identifying the units of the variable. Extra variables are included in the model but are not interpreted. The control provided by other variables is ignored.
Sophisticated analysis includes a reproducible file that combines code to produce statistical results, the actual statistical results themselves, and text interpreting those results in one easy-to-use place. Re-compiling or re-running the statistics produces the same results, even when run on other people’s computers. Includes all relevant calculations in an way that is easy to match up with the writeup, and possibly includes additional relevant statistics with explanation that didn’t make it into the paper, but does not include irrelevant analyses.
Competent analysis can be reproduced, but not necessarily easily. A script or set of instructions exists to reproduce the analysis, and it works correctly on the computer of the analyst (but, unknown whether it produces the same results on other computers with possibly different settings). The output includes all relevant analyses, but requires looking through the file to match up results with the writeup.
Not yet competent analyses cannot be easily reproduced; they require the analyst to figure out how to run the analysis again and do not document exactly how the statistics were calculated the original time. They may also include additional incorrect or irrelevant analyses in the reproducible file without labeling these analyses as potentially misleading.
Sophisticated writeups make a clear argument that is backed up with evidence, and are clear and easy to understand. They are of a professional quality; they contain no spelling errors and are gramatically correct. All tables and graphs are easy to read (including appropriate font size) and all numbers are clearly explained. Additionally, all tables and graphs include a discussion of what the reader should learn from them. Tables and graphs are explained but not duplicating material in the text. The text completely answers all relevant questions with evidence, but without unnecessary repetition of content or irrelevant text. Organized to clearly make argument. It is writen in first person active voice.
Competent writeups contain a clearly developed ideas in a logical sequence. They use graphs and tables to provide data as appropriate, and they are easy to read and provide relevant data. Very few spelling or gramatical errors, but still easy to understand. Professional appearance. May contain some unnecessary repetition or irrelevant text, but that doesn’t hinder understanding the main point. Numbers are explained clearly and include appropriate units.
Not yet competent writeups are disorganized and difficult to follow; they do not make a clear argument or back up that argument with clear evidence. They can appear unprofessional, with numerous spelling errors or grammatical mistakes. Tables or graphs are confusing and difficult to read (small font, too much in them, etc.). Doesn’t interpret or explain graphs, tables, or statistics but instead leaves it up to the reader to figure out what they mean. It is written in third person passive voice.