The Bristol Observatory

              Incorporated in 1997 by John A. Pandiani, Ph.D., Sociologist 

                                             and Steven M. Banks, Ph.D., Mathematician

Access to Child and Adolescent Mental Health Services.  Journal of Child and Family Studies 14: 3, 431-441 September 2005 (Pandiani, Banks, Simon, Van Vleck, Pomeroy)

Adequate access to child and adolescent mental health services for young people in
high need populations is an important concern of service systems researchers and
program evaluators. We present results of a statewide study of access to community
mental health services for eight populations of special concern. The analysis
relied exclusively on existing databases in conjunction with innovative statistical
techniques to provide comprehensive measures of access to care. Our findings indicate
that access to care varied substantially across special populations, although
children and adolescents in each of our eight “special populations” had greater
access to public mental health services than members of the general population
of the state. The interpretation of the findings and directions for future research
are discussed.

Probabilistic Population Estimation of the Size and Overlap of Data Sets Based on Date of Birth. Statistics in Medicine, forthcoming (Banks and Pandiani). 

Probabilistic Population Estimation is a statistical procedure for deriving unduplicated counts of the number of people represented in data sets that do not include unique person identifiers and the number of people shared by data sets that do not share personal identifiers. Because the procedure relies on anonymous data sets, the personal privacy of individuals and the confidentiality of medical records is protected. This paper describes the mathematics of Probabilistic Population Estimation, and applies the procedure to an important contemporary public policy issue.

Age and Mortality Among Problem Drinkers. Addiction. August 2000 (Banks, Pandiani, Schacht, and Gauvin) 

 

PARTICIPANTS: participants include all white men aged 18-79 treated for alcohol-related problems in community mental health substance abuse programs in Vermont during 1991. The treatment group includes 1,853 service recipients; the comparison group includes 196,443 adult white male residents of Vermont.

MEASUREMENT: Measurement of mortality rates for problem drinkers was based on probabilistic determination of overlap between treatment and vital record data sets.

FINDINGS: Mortality for problem drinkers is greater than the general population in all three age groups. The estimated relative risk of mortality in the oldest age group was lower than the other groups, but substantially higher than found in recently published research. The estimated relative risk of mortality in the youngest age group, which has rarely been addressed in previous research, was higher than the relative risk in the middle age group.

CONCLUSION: The application of a public health research model in which problem drinkers are compared to the general population has potential to inform public policy. In this case, the public health approach identified an elevated risk of mortality associated with problem drinking among older adults that had not been evident in the previous research.

After Children’s Services: A Longitudinal Study of Significant Life Events. Journal of Emotional and Behavioral Disorders (Banks, Pandiani, and Schacht) 

In this article we examined the long-term (three year) treatment outcomes of all adolescents who received services in a statewide system of care. The outcomes included rates of hospitalization, incarceration, and maternity. Because the outcome measures were derived from existing administrative and public health data, outcome data were available for all participants. Results indicate that recipients of services had substantially elevated rates of incarceration and hospitalization (compared to the general population) but that maternity rates in the treatment groups were not different from other young women in the same age group. Interestingly, the amount of elevation of incarceration and hospitalization rates decreased over time.

Utilization of Local Jails and General Hospitals by State Psychiatric Center Patients. The Journal of Behavioral Health Services and Research, November 2000 (Banks, Stone, Pandiani, Cox, and Morchauser)

The idea that the deinstitutionalization of state psychiatric centers has resulted in increased utilization of general hospitals and correctional facilities by people with severe and persistent mental illness is widely held. This hypothesis of transinstitutionalization was tested by examining hospitalization and incarceration rates of people who had been or would be institutionalized in state psychiatric centers in 16 upstate New York counties. The results do not support the hypothesis of transinstitutionalization. Assumptions underlying the hypothesis are examined, potential explanations for the observed patterns are discussed, and areas for further research are suggested.

Using Incarceration Rates to Measure Mental Health Program Performance. Journal of Behavioral Health Services and Research, August 1999. (Pandiani, Banks, and Schacht)

This article introduces a more refined conceptualization of the criminal justice involvement of clients of mental health programs than is evident in the existing literature, and demonstrates a research methodology that provides program administrators with standardized measures of program performance in this area.  The conceptualization f the criminal justice involvement of people served by community mental health programs distinguishes between three distinct areas of concern: (1) program accessibility to people with a history of criminal justice involvement, (2) criminal justice outcome, and (3) quality of program performance.  The methodology makes use of existing data resources to provide a valid and reliable measure of program performance in these three areas.

Caseload Segregation/Integration: A Measure of Shared Responsibility for Children and Adolescents. Journal of Emotional and Behavioral Disorders, Summer 1999 (Banks, Pandiani, and Schacht) 

In this article, we introduce a child-focused measure of the degree to which local systems of care share responsibility for children and adolescents (the Caseload Segregation/Integration Ratio) and apply this measure to three service sectors in each of Vermont’s 10 mental health service areas.  The analysis focuses on caseload overlap among three service sectors: community mental health programs, the state child protection agency, and special education for an emotional or behavioral disability.  Because the three service sectors do not share unique person identifiers, probabilistic population estimation was used to derive unduplicated counts of the number of children and adolescents served.  Caseload integration was found to be positively related to the size of mental health service sector, negatively related to the size of the child protection service sector, and unrelated to the size of the special education emotional behavioral service sector.

A Risk Adjusted Measure of Hospitalization Rates for Evaluating Community Mental Health Program Performance. Administration and Policy in Mental Health, March, 1999. (Pandiani, Banks, Schacht, and Gauvin) 

The authors describe the use of risk-adjusted hospitalization rates to measure community mental health treatment outcomes.  The risk adjustment involves comparing rates of hospitalization subsequent to treatment with rates of hospitalization prior to treatment.  The research uses a probabilistic methodology that reliably estimates caseload overlap by comparing the distribution of the dates of berth observed in data sets to the distribution of dates of birth in the general population.  Findings indicate that risk-adjusted hospitalization rates are substantially different than unadjusted rates.  Half of the community programs in one state consistently achieved positive outcomes in four consecutive years; other programs had mixed results or no change.

Personal Privacy vs. Public Accountability: A Technological Solution to an Ethical Dilemma. Journal of Behavioral Health Services and Research, November 1998. (Pandiani, Banks, and Schacht)

The tension between personal privacy and public accountability produces one of the major ethical dilemmas facing behavioral health program evaluators and service system researchers.  This article discusses the source of this tension and introduces a research methodology that allows program evaluators to fully and equally respect both ethical principles.  This methodology uses contemporary computer and statistical technology in conjunction with aggregated, de-identified information derived from existing databases to provide valid and reliable measures of the performance of treatment programs while it protects the personal privacy of individuals.

Practice Patterns and Hospitalization Rates: A Statewide Program Evaluation. Administration and Policy in Mental Health, September, 1998. (Banks, Pandiani, Gauvin, Reardon, Schacht, and Zovistoski)

 Basic indicators of community support program treatment outcome as well as the degree to which the programs conform to practice guidelines for major depression are evaluated.  Hospitalization rates subsequent to treatment are measured, and the correlation between hospitalization rates and practice patterns is determined.  Data sets that describe outpatient and inpatient services over a 4-year period, but do not include common person identifiers were analyzed using probabilistic population estimates.  Results indicate that there is substantial variation amount the community programs in practice patterns and hospitalization rates, and the two are negatively correlated.

 

The Use of State and General Hospitals for Inpatient Psychiatric Care. American Journal of Public Health, March 1998. (Banks and Pandiani)

 Objectives:  This paper explores the relationship of state hospital and general hospital psychiatric caseloads in a statewide system of care.

Methods:  Probabilistic population estimation was applied to general hospital and state hospital data sets.

Results:  General hospitals provide inpatient psychiatric services to more people than do state hospitals, and a significant number are served in both sectors.  There were notable differences in use patterns related to patient gender and age.

Conclusions:  These results demonstrate that probabilistic methodologies can significantly enhance the value of existing databases for epidemiological research.

 

A Global Measure of Access to Mental Health Services for a Managed Care Environment. (Pandiani, Banks, and Gauvin)

The growing influence of managed care in mental health services has raised important questions about access to services.  This article introduces and demonstrates a global measure of access that is based on the relationship between service utilization and the need for services.  This measure has become practical because of recent advances in measurement technology that provide more valid and reliable estimates of the prevalence of mental illness in general populations and of the number of people who receive mental health services across service sectors.  The methodology is used to produce a report card type profile of access to inpatient mental health services (in state, general, private, and veterans hospitals) by residents of one state.  The global measure can provide a powerful and efficient tool for monitoring and comparing the impact of managed care plans on access to mental health services.

Consumer Satisfaction and Treatment Outcomes: Dissatisfaction with Mental Health Services and Incarceration after Treatment. (Pandiani, Schacht, Banks) 

Consumer satisfaction is increasingly used to measure community mental health program performance. The relationship between consumer satisfaction and treatment outcomes is only beginning to be understood. This paper adds to this understanding by reporting the results of an assessment of the relationship between consumer evaluation of community mental health services and incarceration subsequent to treatment in a statewide system of care. Results indicate that satisfaction with services is related to incarceration subsequent to treatment, with satisfied consumers having lower incarceration rates. These results support the utilization of self-reported consumer satisfaction as a measure of mental health program performance.

 

Approaches to Risk Adjusting Outcome Measures Applied to Criminal Justice Involvement after Community Service. (Banks, Pandiani, and Bramley). 

The ethic of fairness in program evaluation requires that measures of behavioral health agency performance be sensitive to differences in those agencies' caseload composition. The authors describe two traditional approaches to the statistical risk adjustment of outcome measures (stratification-weighting and pre-post measurement) that are designed to account for differences in caseload composition, and introduce a method that incorporates the strengths of both approaches. Procedures for deriving each of these measures are described in detail and demonstrated in the evaluation of a statewide system of community based behavioral health care programs. This evaluation examines the degree to which service recipients get into trouble with the law after treatment. Three measures are recommended for inclusion in outcome oriented "report cards" and the interpretation of each measure is discussed. Finally, the authors suggest formats for graphic and tabular presentation of the risk adjusted evaluation for sharing findings with diverse stakeholder groups.

 

Bed Closures and Incarceration Among Users of VA Behavioral Health Services in Upstate New York. Mental Health Services and Research, October 2000 (Rosenheck, Banks, Pandiani, Hoff) 

Objective: This study examined incarceration of users of Department of Veterans Affairs (VA) mental health services in 16 northeastern New York State counties between 1994 and 1997 -- a time of extensive bed closures in the VA system -- to determine whether incarceration rates changed during this period. 

Methods: Data was obtained for male patients who used inpatient and outpatient VA mental health services between 1994 and 1997 and for men incarcerated in local jails during this period.  For comparison, services used and incarceration data were obtained for all men who received inpatient behavioral health care at community general hospitals and state mental hospitals between 1994 and 1996 in the same counties.  probabilistic population estimation, a novel statistical technique, was employed to evaluate the degree of overlap between clinical and incarceration populations without relying on person-specific identifiers. 

Results: Of all male users of VA mental health services between 1994 and 1997, a total of 15.7 percent - 39.6 percent of those age 18 to 39 years and 9.1 percent of those age 40 years and older -- were incarcerated at some time during the period.  Dual diagnosis patients had the highest rate of incarceration (25 percent), followed by patients with substance abuse problems only (21 percent) and those with mental health problems only (11 percent).  The rate of incarceration among male patients hospitalized in VA facilities was lower than among men in general hospitals or state hospitals (11.6 percent, 23 percent, and 21.7 percent, respectively), but was not significantly different.  No significant increase occurred in the annual rate of incarceration among VA patients from 1994 to 1997 (3.7 percent to 4 percent), despite extensive VA bed closures during these years.

Conclusion: Substantial proportions of mental health system users were incarcerated during the study period, especially younger men and those with both substance use and mental health disorders.  Rates of incarceration were similar across health care systems.  The closure of a substantial number of VA mental health inpatient beds did not seem to affect the rate of incarceration among VA service users.  (Psychiatric Services 51:1282-1287, 2000)

 

Objective: This study of users of Department of Veterans Affairs (VA) behavioral health services examines changes in their use of non-VA inpatient services from 1994-1997, a period of extensive VA bed closures 

Methods: Data on service use were obtained on New York State residents who used VA behavioral health services, and on residents who were hospitalized for behavioral health care at community General Hospitals and State Mental Hospitals. Probabilistic Population Estimation, a novel statistical technique was employed to evaluate the degree of overlap between VA mental health and non-VA inpatient populations without relying on person-specific identifiers. 

Results: Annually, on average, 6% of VA behavioral health service users were hospitalized in either General Hospital in New York State or State Mental Hospitals: 4% of psychiatric patients, 10% of substance abuse patients and 11% of dually diagnosed patients. Overall rates of cross system use did not change significantly from 1994-1997. 

Conclusion: Cross system use of inpatient services is limited, and was not affected by the closure of one-third of all VA mental health beds, most likely because of an increased supply of VA outpatient services.

 

This study of users of Department of Veterans Affairs (VA) behavioral health services examines changes in their use of non-VA inpatient services from 1994 to 1997, a period of extensive VA bed closures. Data on service use were obtained on New York State residents who used VA behavioral health services and on residents who were hospitalized for behavioral health care at community general hospitals and state mental hospitals. Probabilistic population estimation, a novel statistical technique, was employed to evaluate the degree of overlap between VA mental health and non-VA inpatient populations without relying on person-specific identifiers. Annually, on average, 6% of VA behavioral health service users were hospitalized in either general hospitals in New York State or state mental hospitals: 4% of psychiatric patients, 10% of substance abuse patients, and 11% of dually diagnosed patients. Overall rates of cross system use did not change significantly from 1994 to 1997. Cross-system use of inpatient services is limited and was not affected by the closure of one-third of all VA mental health beds, most likely because of an increased supply of VA outpatient services.

 

Elevated Risk of Being Charged with a Crime for People with a Severe and Persistent Mental Illness. (Pandiani, Banks, Clements, and Schacht) 

This paper examines the criminal justice involvement of people served in a statewide system of care for people with severe and persistent mental illness, a concern of both the mental health and the law enforcement communities. Rates at which service recipients were charged with a crime were determined using Probabilistic Population Estimation to measure the number of people shared between anonymous data sets. Results indicate that members of the treatment group were significantly more likely to get into trouble with the law than members of the general population. Elevated risk is substantially greater for women and for older treatment recipients.

Probabilistic population estimation of the size and overlap of data sets based on date of birth.  Statistic in Medicine Volume 20, Issue 9-10, May 2001

Probabilistic population estimation is a statistical procedure for deriving unduplicated counts of the number of people represented in data sets that do not include unique person identifiers and the number of people shared by data sets that do not share personal identifiers.  Because the procedure relies on anonymous data sets, the personal privacy of individuals and the confidentiality of medical records is protected.  This paper describes the mathematics of probabilistic population estimation, and applies the procedure to an important contemporary public policy issue.  

 

Cross State Comparison of Mortality for Mental Health Service Recipients.   (Pandiani, Banks, Bramley, Moore).

Recent papers in this journal evidence growing interest in system level comparisons of mortality rates for people with mental disorders.  Important concerns have been raised as well.  This paper presents statewide mortality rates for people with severe mental illness served in public mental health systems of care in Vermont and Oklahoma.  In both states, mortality rates for younger service recipients, but not older service recipients were significantly elevated compared to the general population.  Based on their experience, the authors propose guidelines for standardizing the computation and presentation of system level comparison of mortality rates.

 

Measuring Access To Mental Health Care: A Multi-indicator Approach to Program Evaluation.   (Pandiani, Banks, Bramley, Pomeroy, Simon).

Access to services in publicly funded systems of care is a long standing concern of program evaluation.  This paper demonstrates a multifaceted approach to measuring access to care that focuses on both general populations and special populations believed to have an elevated need for services.  The approach relies on existing data sets (no new data collection) in conjunction with innovative statistical techniques.  A set of generic decision points involved in studies of access to care are identified and discussed.  Finally, the broader utility of this approach for program evaluation is discussed and future directions for research on access to mental health services are identified. 

Caseload Segregation/Integration and Treatment Outcomes for Children and Adolescents.  (Banks, Pandiani, and Schacht)

In this article we explore the relationship between the degree to which local systems of care share responsibility for children and adolescents (measured by the Caseload Segregation/Integration Ratio) and four service delivery outcomes in one state over a four year period. Caseload segregation/integration is measured for community mental health agencies, child protection and juvenile justice programs, and special education programs for emotional and behavioral disorders. Services outcomes under examination include referral for intensive residential treatment and hospitalization for behavioral health care (for all children), incarceration (for boys) and maternity (for girls). Results indicate that greater caseload integration is associated with less referral for residential treatment, and higher rates of incarceration. The interpretation of these findings is discussed and further research is suggested.  

 

Elevated Risk of Arrest for Veteran’s Administration Behavioral Health Service Recipients in Four Florida Counties (Pandiani, Rosenheck, and Banks)  

This paper examines the relative contribution of mental and substance abuse disorders to criminal justice involvement by examining the relative risk of arrest for three groups of adult male recipients of VA behavioral health care services. These groups include men served for both substance abuse and mental health, for only substance abuse, and for only mental health. The relative risk of multiple offences is compared to relative risk of a single offence for each group. Results indicated that relative risk of multiple arrests for the dual diagnosis group is substantially greater than for either of the single diagnosis groups, and greater than the relative risk for recipients of non-behavioral health services. Relative risk of arrest for recipients of only mental health services is no different than the relative risk for other veterans living in the region under examination.

Atypical Anti-Psychotic Medications and Criminal Justice Outcomes (Pandiani, Pomeroy, and Banks)

OBJECTIVE:  To measure the impact of beginning atypical medication on being charged with a crime.     

RESULTS:  Fewer adults served by community mental health programs for people with severe and persistent mental illness got into trouble with the law after beginning atypical anti-psychotic medication than before.  This impact was not uniformly felt, however.  Men experienced decreased criminal justice involvement, but women experienced increased criminal justice involvement.  

METHOD: This statewide study relied entirely on existing databases in conjunction with probabilistic statistical techniques that measure cross service sector involvement without reference to personally identifiable information.  

CONCLUSIONS:  The finding that the impact of medication is mediated by client age and gender has important implications for future research and, potentially, for clinical practice. 

Educational Test Scores for Measuring Children's Mental Health Programs Performance (Pandiani, Simon, Banks)   

Standardized educational test results are becoming an important part of the children's services data infrastructure.  This paper reports the results of the first statewide application of these test scores to the evaluation of community mental health children’s services programs.  Standardized test data were used to compare the school participation and school performance of service recipients to other children.  School participation rates for services recipients varied substantially among community programs but all were significantly lower than the participation rates for other young people.  School performance for services recipients also varied substantially among community programs but school performance for service recipients in some local programs were not significantly different than the performance of other young people.  The utility of these findings for program monitoring and evaluation will be discussed.

 

Toward an Operational Definition of “Personally Identifiable Information” (Banks and  Pandiani, The Bristol Observatory) 

OBJECTIVES: To provide an operational definition of “personally identifiable information” that will allow health researchers and data archive administrators to make reasonable decisions about access to data. This operational definition should be sensitive to both the personal privacy of individuals and the need of program administrators, health care providers, payers, and the public to obtain information on the performance of hearth care providers and systems of care. 

METHODS: Statistical procedures derived from probability theory are used to determine the population size necessary to achieve specified levels of identifiability. These statistical procedures are applied to the problem of determining the size of a community in which the combination of date of birth and gender may be considered personally identifying information.  

The determination of the probability that more than one person in the population will share a birthday (month and day) is provided by the solution to the classic birthday problem. Similar logic provides the probabilities related to the sharing date of birth (month, day, and year) and gender.

The probability that a specified person will share his or her date of birth and gender with another person in a population is calculated as the average probability of sharing and as the minimum probability of sharing.

RESULTS: Date of birth and gender tend to be personally identifiable in communities of less than 2,600 people. When populations reach 72,000, false identification is more likely than correct identification of individual people.

CONCLUSION: Data sets that include the date of birth and the gender of individuals in conjunction with county of residence should rarely be considered personally identifying. The statistical procedures described in this presentation can be applied to any set of personal characteristics when the distribution of these characteristics in the population is known. This information provides a rational basis for making decisions about access to data sets.

The ability to know the degree to which date of birth and gender should be considered to be personally identifying is particularly important in light of the ability of Probabilistic Population Estimation to measure critical treatment outcomes. Probabilistic Population Estimation is a statistical technique for measuring the number of people represented in both of two anonymous data sets (e.g. a treatment data set and a mortality data set) based on the distribution of date of birth and gender in the data sets (Banks and Pandiani, Statistics in Medicine, in press). This methodology is particularly important in this era of heightened concern about the protection of personal privacy (Pandiani and Banks, Journal of Behavioral Health Services and Research 25:4, 456-463).  

Impact of Multi-Agency Employment Services on Employment Rates.  Community Mental Health Journal (Pandiani, Tracy, Simon, Banks) 

Adults with severe and persistent mental illness who received employment services through mental health and/or vocational rehabilitation programs had higher employment rates than individuals who did not receive any employment services.  Individuals who received services from both programs had significantly higher employment rates than individuals who received services from only one program.  Results indicate that employment services had a greater relative effect on older clients and clients with a schizophrenia diagnosis than on other individuals.  This statewide study relied exclusively on analysis of administrative/operational databases that provide the employment rates for both recipients of vocational services and other clients. 

 

ACCESS TO CHILD AND ADOLESCENT MENTAL HEALTH SERVICES(Pandiani, Banks, Bramley, Simon, Pomeroy)

Access to services in publicly funded systems of care is a long-standing concern of program evaluation.  This case study describes a multifaceted approach to measuring access to care that focuses on both general populations and special populations believed to have an elevated need for services.  The approach relies on existing data sets (no new data collection) in conjunction with innovative statistical techniques.  A set of generic decision points involved in studies of access to care are identified and discussed.  Finally, lessons learned from this project and directions for future research on access to mental health services are identified and discussed.

 

Involvement in the Criminal Justice System Among Recipients of Mental Health Services After September 11. (Pandiani JA, Knisley MB, Banks SM, Simon M, Blackburn P) 

Objective: This study examines relative risk of arrest for mental health service recipients in Washington D.C. during the 23 months before and the 13 months after September 11, 2001 in order to determine if any change in criminal justice involvement followed the terrorist attacks.     

Methods:  Analysis of anonymous data sets provided by the Mental Health and Police Departments provided measures of risk of arrest (relative to the general population) for the approximately 5,000 people served each month.  Interrupted time series analysis was used to measure change in relative risk during the 36 month study period. 

Results:  There were significant increases in relative risk of arrest following September 11, 2001 for male and non-white young adult mental health service recipients.  Significant change was evident for eight of the 12 age/gender/racial  groups used in this analysis.

 Conclusions:  Disaster planning and response should include attention to criminal justice involvement for mental health service recipients.  Future research should focus on the nature of the relationship between terror and arrest for service recipients.