Prepared by Brooks Ambrose

1 Overview

RDADAR is a rubric scoring system intended to give you substantive feedback on your Week 8 case study midterm. The goals of the feedback are:

To show the strengths of your submission.
To highlight areas of improvement.
To provide an opportunity to reflect on how your project ideas fit into the RDADA framework.

RDADAR is not a tool to evaluate or assign your grade for the midterm. Rather we hope it helps you consider ways to extend and improve your work. We also hope that it can serve as a touchstone for you to discuss your ideas with your peers and your instructor.

Note that RDADAR is biased toward thoroughness or completeness according to the particular dimensions of quality outlined below. If you chose to take a “deep dive” into one dimension at the expense of others, for instance if you wrote a concept piece and did not trouble yourself with the details of the empirical research, then your overall RDADAR score will be low even if you did high quality work. These criteria are not exhaustive of what can be considered good work, but they do provide a template for what a balanced data science project proposal would include.

1.1 Assessment

The assessment criteria are organized after the three main sections of the course: Decision Making, Research Design, and Conveying Findings. In addition, several features of organizational science are separated from Decision Making and placed into an Organizational Design category, and a Scope category considers broader features of making an impact through data science, a theme that runs through each section of the course. Criteria are listed and described below, with any comments.

In addition to the categories defined in the rubric, a statistical approach to clustering the criteria was performed that identified five emergent categories of highly correlated criteria. Though these categories merely combine the underlying criteria in a different way, they do provide a slightly different conception of data science quality. These emergent categories are described under the section of the most relevant dimension. They are framing, measurement, implementation in context, social concept, and focused realism.

The scores presented below are for the submission with the file name Sample-Submission-Random-Draw. The numerical prefix is the index number used for anonymous scoring. Please refer to this index when you approach us with questions or comments.

2 Individual Criterion Scores

For each rubric criterion (29 variables) and each submission (n = 27) a score was given from 0 to 3, where 0 indicated that the criterion was missing, and 1, 2, and 3 meant low, average, and high quality performance respectively. Scores within a category were then summed and the distributions are reported below. The absolute values of these scores are not very meaningful and are suppressed in the plots; we believe it is more useful to see where you fall in the distribution relative to your peers. The following chart displays your scores for each criterion as well as its deviation from the class average. The criteria at the top of the chart are the ones you performed the best at relative to your peers, while those toward the bottom are areas to consider improving. See below for descriptions of each criterion.

3 Aggregate Scores by Category

Rather than conveying arbitrary absolute scores, we present your score as falling within the class distribution. A consequence of this “curve” is that 50% of you will be below the median (which should not be a surprise!). Being at the bottom of a distribution does not necessarily mean that your work was of poor quality, however it may indicate that your peers are meeting a higher standard. Where possible the distribution is divided into quintiles and the quintile where your score fell is highlighted in gold. Where quintiles cannot be plotted the raw composite score is given instead, again with your score highlighted in gold. The following illustrates how to interpret the plots.

Notes on reading the histograms:

Quantile histograms will produce uneven intervals. Quintiles by definition should each contain 20% of the data, or about 5.4 observations.
The mean for each distribution is plotted in red. Compare the bin highlighted in gold to the red line to get a sense of how you performed compared to the average.
Quantile plots are shown both with a rug and a kernel density estimate, each indicating the continuous distribution underlying the quantiles. Because the bar charts represent the data literally they do not include the density illustrations.
Depending on skewness, the tail quantiles may be very wide and contain scores that are far apart. We hope that presenting your score as a quantile instead of an exact number discourages you from fixating on whether you were the highest or lowest scorer, which would be based on an unreliable and error prone measure anyway. Quantiles represent a more stable and robust assessment, but you should feel free to consider that your score might fall in an adjacent quantile. A relatively low n of 27 and errors of judgment on the part of the coder would undoubtedly result in some float and bounce in your numbers if the submissions were reassessed.

3.1 Total Score

Your total score is a good indication of overall quality relative to your peers. No matter where you find yourself, working on your biggest weaknesses first is a good strategy for strengthening your position on the next project proposal.

The following sections break down your total score by category, including the inductively chosen categories. Due to time constraints, not all scores are justified with a comment, and these will list “NA”. You are encouraged to request a justification for any score.

3.2 Decision Making

DECISION MAKING	Criterion	Scr	Comment
Problem	Was the problem motivating the study clearly explained?	2	NA
Strategy	Was a high-level or long-term goal articulated to address the problem?	2	NA
Tactic	Were tactics to achieve the strategy, including but not limited to the proposal on the table, discussed?	3	NA
Client	Was it clear who the clients were?	1	NA
Decision	Was it clear who the decision makers were, and was the decision precisely stated?	2	NA
Salience	Was it clear why the problem was important to the client, and why the client would be motivated to support the proposal?	2	NA

3.2.1 Framing

Framing is a combination of RQ, problem, strategy, and intended impact. This refers to a close alignment of problem and research question. High scores here indicate a great clarity about what is to be done and why that helps the reader follow the more granular details of the case study.

3.3 Research Design

RESEARCH DESIGN	Criterion	Scr	Comment
RQ	Was the problem usefully stated as a research question?	1	NA
Conceptualization	Were the details and ideas necessary to understand the problem explained, including assumptions about causal relationships?	1	NA
Operationalization	Were the concepts and relationships carefully matched to valid empirical indicators?	1	NA
Unit	What were the “thing” or “things” you were trying to study? Were the units directly observable, and if not, how did you plan to study them?	0	NA
Metrics	Were the variables or features you wished to observe about your units clearly described, including the possible values they might take?	2	NA
Instrumentation	Were the technologies, techniques, or procedures used to measure your metrics explained?	2	NA
Method	Once measurements were taken, was it clear which statistical tests or interpretive techniques would be appropriate for analyzing the data? Would these methods allow you to answer the research question?	3	NA
Results	Were the real or predicted outcomes of your methods presented or illustrated?	2	NA

3.3.1 Measurement

Measurement is a combination of metrics, and instrumentation. This highlights careful attention to data gathering.

3.4 Organizational Design

ORGANIZATIONAL DESIGN	Criterion	Scr	Comment
Implementation	Was it clear how the research design would be executed and by whom?	3	NA
Organizational Context	Was the firm or organization described, including the relevant roles and relationships affecting the data scientist and decision makers?	0	NA
Politics	Was there attention to organizational tensions or interests that might facilitate or block the implementation of the research design?	2	NA
Biases	Were cognitive or culture assumptions or expectations that might influence the research design considered?	1	NA
Ethics	Were questions of legality, fairness, and possible adverse or inequitable outcomes considered? Are there professional standards or formal codes of conduct that need to be addressed?	2	NA

3.4.1 Implementation in Context

Implementation in Context is a combination of organizational context and implementation. This highlights special attention to the practical problems and opportunities facing the project.

3.5 Conveying Findings

CONVEYING FINDINGS	Criterion	Scr	Comment
Communication	Did you design media or explain your plan for conveying findings to the decision makers and relevant stakeholders?	1	NA
Translation	Did you communicate your results in a way that would be easy for a non-expert to understand?	3	NA
Inference / Recommendation	Did you clearly communicate the meaning or significance of your findings, possibly including recommendations for taking actions or making decisions?	1	NA

3.6 Scope

SCOPE	Criterion	Scr	Comment
Ambition	Does the project suggest a big effort and a bigger payoff? Does it push the boundaries of what has been done before?	3	NA
Realism	Is the project tractable?	1	NA
Focus	Did the project have a strong armature, or did you consistently address each detail to solving the problem?	3	NA
Stakeholders	Did you discuss parties besides the decision maker and client who would be affected by the project or who have an interests in its success or failure?	2	NA
Reach	How many people, firms, or industries is the project likely to impact? Is it likely that effects would multiply beyond the client?	2	NA
Intended Impact	Are you clear about what consequences you intend to have as a result of implementing the project?	3	NA
Unintended Impact	Did you consider the consequences that you did not plan for or expect?	1	NA

3.6.1 Focused Realism

Focused Realism is a combination of focus and realism, this dimension captures whether you were able to create a clear and concrete image of a “doable” project.

4 General Comments

5 Feedback to the RDADA Team

Thank you for reviewing this assessment tool. Please report bugs or address your questions and comments to brooksambrose@berkeley.edu with “RDADAR” in the subject line. We are very interested in knowing what is and isn’t useful about this tool. We are also curious to know whether you had different criteria in mind when preparing your submission. Part of the goal of this exercise is to share our standards and judgement of quality explicit, and hopefully to arrive at a community concept of what separates good from bad data science.

The RDADA team would like to acknowledge Peter Norlander for inventing RDADAR.

RDADAR

Fall 2015 Midterm Assessment