In this article, the authors examine a rubric used to assess students’ writing in a large-scale testing program. They present empirical evidence for the existence of a potentially widespread threat to the validity of rubric assessments that arose due to design features. The research casts doubt on whether rubrics with structurally aligned categories can validly assess complex skills.