Performance Measures

Performance measures for criminal justice programs

For decades, responsible elected and appointed officials at all levels of government have sought measures of the efficacy of programs so as to permit the development of a public version of return on investment (ROI) even though there is no single acceptable measure of merit in public services.  Defining performance outcomes that are measurable is particularly elusive in the criminal justice field.  However, the need is great to define a way to help executives and legislators make judgments on which programs to fund in a rational way that is driven by measurable performance. 

The impact on society of investments in criminal justice process improvement ought to support the responsibility of government to provide for public safety and the effective administration of justice.   Federal funding support for new programs that have been shown in some way to be effective has been based often on intuitive conclusions or anecdotal evidence and seldom has there been reliable data let along measures that have influenced spending decisions. 

Legislators have often assumed that the funding of particular programs will reduce crime and therefore the programs are worthy of the investment.  But when it comes time to evaluate such programs, there appears to be no particular correlation between the reduction of crime and the investment in the program.

The concept of evaluating programs in terms of the degree to which they reduce crime is both impossible and largely useless, for the following reasons.

1. We do not have valid and consistent data on the amount of crime in America. 

The only national data on the amount of crime that occurs is the FBI’s Uniform Crime Report (UCR) program.  Starting as early as the Wickersham Commission in 1937 researchers have argued that the Uniform Crime Reports generated by local agencies and aggregated by the FBI into national statistics are largely useless for providing an analytical basis for measuring any particular program or even understanding the nature of crime in the United States.  The Commission observed that many police departments simply do not report crime data, and in the period of leading up to the Presidential Crime Commission of 1967 little progress had been made to improve national reporting.  

Thanks to efforts made by the FBI, funding allocated to the Bureau of Justice Statistics, and the introduction of Statistical Analysis Centers in states, the percentage of police organizations reporting to UCR went from the 70 percentile to the mid-to-high 90’s during the decades following the President’s Crime Commission, so there is a high level of reporting today.   However, police can only tabulate crimes reported to the police, and the bulk of crimes are reported primarily for insurance purposes, meaning that only those citizens who have insurance are motivated to report crimes other than violent crimes against persons.  The national Victimization Surveys begun by the Bureau of Justice Statistics were an attempt to discover by survey methods what crimes occurred regardless of the extent of reporting, and to formulate estimates based on the survey data.

It has also been well established that cultural attitudes impact the extent to which citizens choose to report crimes to the police.   Only to the extent that the public or specific cultural groups believe that some positive result will come from the action of making a crime report, or for insurance purposes, do people inform the police.   Where there is a belief that nothing will result from reporting, people simply do not tell the police about crimes. 

The Uniform Crime Reports suffer from another problem that has frustrated researchers and decision makers for many years.   Only the most serious offense related to the event of the crime is counted in UCR, and this policy masks countless associated offenses that may have occurred.   As the FBI has attempted to introduce the National Incident Based Reporting System (NIBRS) with its ability to count multiple offenses, the total amount of crime will increase only because of the increased accuracy of counting crime (offenses). However, many states and major cities have been reluctant to embrace the NIBRS reporting standards and therefore the effect of this change is difficult to measure.

The efforts to improve reporting by police agencies, the introduction of community based policing which may improve the reporting of crime, the revision of the counting methodology can all serve actually to increase the amount of crime measured by such programs so that the attempt to measure program performance in terms of the extent to which crime is reduced becomes overshadowed by the increases due to other factors.

2. Variations in the crime rate are influenced more by external factors than any given program or policy.

Many studies of the crime rate variations have concluded the at least on a national level that the variation of the crime rate is influenced mainly by demographic changes.   Crime is mostly committed by young people, and as the percentage of the population who are in the ages prone to engage in crime changes, so does the crime rate.  At the very least, this factor has to be accounted for in any attempt to use crime data as a program performance metric, and models for doing so have yet to be developed.

Studies have also shown that the amount of crime is highly correlated to the number of police officers.  As the COPS program moved to increase the number of police officers, it is likely to have resulted in a higher reported number of crimes.   This factor was evident in the sixties and seventies as more police were assigned to narcotics work and the number of crimes rose as a result.   The number of recorded “crimes against society” and “victimless” crimes is clearly influenced by the law enforcement efforts applied to these areas.

Other social factors, such as the ethnic composition of the population, economic conditions, joblessness, homelessness have all been shown to have a significant influence on the crime rate irrespective of programmatic attempts to achieve crime reduction.  These social factors affect the extent to which crime is actually reported to the police as well as the amount of crime that occurs.

Because of the many variables that have been shown to influence crime rates, it is very difficult if not impossible to conduct the kind of evaluative research that would properly control all of the variables and isolate the influence of any particular program.  The normal scientific methods of testing hypotheses by the use of control groups to control for the many variables is operationally challenging.  Thing simply change too fast, and the highly personal impact of varying the service provided to a class of victims for research purposes is not well regarded by either elected officials or the public they serve.

It could be argued that the better metric of performance for any given program should be the relative reduction in the amount of crime that would have been committed in the absence of the program or policy.   The lack of an accepted model that would take into consideration the large number of factors that influence the crime rate also makes it impossible to estimate the possible avoidance or reduction in crime attributable to a single program.    There is no method for estimating the amount of crime that has not occurred.

3. Crime reduction is not the responsibility or direct objective of many justice agencies and programs.

Even though the public expects that the objective of all criminal justice programs is the reduction in crime, many agencies in the criminal justice “system” do not have a direct charter/mission to reduce crime.  The activities of the police in apprehension, the prosecutor and courts in adjudication, and the correctional institutions in incarceration all have objectives that are unique to their respective constitutionally-derived duties.   While it is loosely assumed that the effective administration of justice and subsequent incarceration of persons justly convicted will have an impact on crime, this assumption is based on intuition and not data.   It is quite clear to the personnel in their respective justice agencies that their objectives are in line with the duty and obligations of their respective roles, and not specifically on crime reduction.  

The courts are currently struggling with the development of performance measures and none of the proposed measures has anything to do with crime reduction.  Courts are concerned about fulfilling their constitutional role in adjudication under a loosely coupled notion that if offenders are justly convicted and dealt with by state of local corrections and supervision agencies that there will be some impact on public safety, but the fundamental objective of courts and the proposed performance measures are about doing a better job of adjudication.   

The administration of justice from arrest through sentencing derives from the government’s responsibility to respond to the commission of crimes as proscribed by law and public policy rather than directly in support of crime reduction.   Even the models of community based policing are designed to engage the community with the police to foster better communications with the community.  There is an assumption that such activities will ultimately have crime reduction impacts, but little proof.   Yet police, elected officials, and the general public generally support concepts of community based policing just because it is a better way of working together.   This model also tends to improve crime reporting (and therefore potentially increase the crime rate). 

To the extent that the causes of crime are factors outside of the control of agencies engaged in law enforcement and the administration of justice, the ability of these agencies to influence the crime rate no matter what they do is questionable. 

Alternative measures of performance

Criminal justice programs and projects that are funded to make fundamental changes in the way that law enforcement and justice agencies perform their statutory duties so as to increase the service to the public or to decrease the cost of providing such services generally should be linked to their impact on society.   Impact is a function of the extent to which a particular program can achieve change in ways the public expects or policy-makers envision.  Program objectives should be linked to strategic plans and show evidence of success with respect to those plans.   There are a set of strategic objectives that could be identified that could be defined and used as a measure of success.   The 1967 Crime Commission identified a unified strategy specific to each component of the criminal justice system, and perhaps these strategies should be updated.   In the midst of a war on terrorism, there are certainly contemporary issues that affect the strategies of four decades ago.  

Given the suboptimization of justice agencies with respect to larger goals of society, and the division of responsibility through the largely adversarial system of justice that we endorse, program performance evaluation is best defined with respect to the extent to which a program supports national policy and priorities.   Some examples:

Many proposed programs and projects in law enforcement could be better measured in terms of the increase in the clearance rate for reported crimes.  To the extent that we charge law enforcement with the apprehension of offenders, projects which seek to improve the likelihood of apprehension and successful prosecution, such as DNA testing and a national DNA database, can be shown to be directly linked to improved clearance rates and more just adjudications.   Also in law enforcement, the public continues to desire to reduce the fear of crime, and programs that improve communications with the community will have this result. 

Projects that have as their objectives the speedier handling of criminal charges in the courts can have a direct impact on the costs of the judicial process as well as the likelihood of convictions and subsequent incarceration where appropriate.  Measuring the percentage of cases that are reversed on appeal can provide indications of the accuracy and equity of lower court proceedings.

Programs should be measured with respect to the factors under the control of the implementing agency.  It is of little merit to measure the performance of the police in terms of actions under the control of the courts. 

In summary, there is no single measure of performance that helps guide program development and funding decisions.   Instead, there is a need to define a series or system of such strategic actions that support policies to which the public and its elected officials subscribe.  Such measures should address the intentions of society in improving the quality of justice in America as well as making our communities safer.

Paul Wormeli

Executive Director

IJIS Institute

May 2004

© Copyright 2005 Paul Wormeli. Last update: 3/16/2005; 9:25:32 PM.