One afternoon not long ago, I sat entering notes into a patient’s medical record. She was in her forties, and her labs showed anemia. The causes of anemia range from menstruation to cancer, and so pinpointing the correct underlying diagnosis is critical. Physicians are trained to formulate a full roster of possibilities, known as the differential diagnosis, and then to work down the list systematically. We’re taught to cast a wide net—celiac disease, parasitic infections, thalassemia, lead poisoning, liver disease, B12 deficiency, myeloma, sickle-cell disease, G6PD deficiency—because you’ll never make a diagnosis if you haven’t included it in your differential.
But I hesitated before entering my differential into the computer system. Should I include the more serious possibilities, even though they were much less likely? In the past, I wouldn’t have thought twice about it, as the chart served primarily as a tool for the medical team to communicate among ourselves. But a new law, the 21st Century Cures Act, had recently been fully implemented, making medical records open to patients by default, in real time, including doctors’ notes. My in-box was already jammed with panicked messages from people convinced that they had catastrophic illnesses, based on minuscule lab discrepancies and panic-inducing Google searches. How would my patient react to seeing my ruminations about possible colon cancer or duodenal ulcer in the note?
One of the earliest known medical records—an account of wound treatment—was written on papyrus in Egypt, around 1600 B.C. Its intended audience seems to have been fellow-healers, who could learn from it. More than a thousand years later, Hippocrates helped develop the concept of the medical-case history, which was likely geared toward educating and informing other practitioners. This contributed to a physician-centric view of the medical record, which has endured. Into the nineteen-seventies, patients in most U.S. states could obtain their medical records only with the help of a lawyer. In 2000, the HIPAA Privacy Rule—which clarified how the health-care industry should implement the Health Insurance Portability and Accountability Act of 1996—required organizations to disclose most kinds of medical information to patients upon request. But health-care organizations were under no obligation to eliminate the paperwork, fees, or surly clerks that stood between patients and their records.
As medical records inched from paper to computer during the nineteen-nineties, a few institutions began experimenting with patient portals—Web sites for patients to access their records. These sites, which began appearing more widely in the two-thousands, were clunky to use, and there wasn’t much in there beyond appointments and some test results. But, in 2009, as part of its attempt to drag the U.S. economy out of recession, the federal government offered incentive payments to medical organizations that could demonstrate “meaningful use” of electronic medical records. Requirements for the funding came to include the ability for patients to “view, download, and transmit” their medical information. With money now on the table, patient portals proliferated.
In 2010, Jan Walker and Tom Delbanco, who have backgrounds in nursing and internal medicine, respectively, founded a research collaborative called OpenNotes, based at Beth Israel Deaconess Medical Center in Boston. This initiative went a step further and experimented with allowing patients to read the actual notes that their doctors were writing. Patients in the initial studies were overwhelmingly enthusiastic about the idea. For physicians, this was a radical change: lab values and radiology reports were neutral information, but notes felt personal. The Cures Act went into full effect in the spring of 2021, granting patients access to their records by default, so that they didn’t have to request it. In the brief lull between COVID-19’s soul-crushing winter and the start of the deadly Delta wave, the medical profession abruptly found itself staring into a two-way mirror.
Transparency has always been seen as a hallmark of honesty and integrity. The logic is familiar: democracies aspire to be transparent, but dictatorships are opaque; faithful spouses are guileless, while philanderers lie; reputable businesses operate in the open, but shady operations literally draw the shades. Collectively, we’ve embraced Louis Brandeis’s dictum that sunlight is the best disinfectant. We demand transparency in government, charitable institutions, nutrition labels, and middle-school grading rubrics. The medical record should be no different.
And yet, in writing the medical note for my patient with anemia, I felt an awkwardness creeping in. There was something disquieting about knowing that my every word might be scrutinized. The academic purist in me pressed for a thorough differential and a full accounting of my thought process—to avoid error, but also to accurately inform the next physician about the current state of affairs. I worried, however, about downstream angst, both for me and for the patient. She had been transparent with me, sharing her story, her body, her blood chemistry. She’d answered personal questions about her medical history, sexual partners, drug and alcohol use, and dietary habits. Still, I hesitated when it came time for me to be fully transparent with her.
In the end, I dashed off something like “anemia eval” —a pathetic skeleton of a note that I wouldn’t have tolerated from a medical student—and tore off the exam-table paper in preparation for the next patient. I felt rushed, annoyed, and frankly disgusted at my dumbing down of the chart. And I couldn’t tamp a rising coil of unease. Why had a system designed to encourage transparency resulted in my drawing the curtain?
In 2008, Ethan Bernstein, an associate professor at Harvard Business School, travelled with a team of students to a vast mobile-phone-manufacturing plant in southern China. Managers scanned stadium-sized production floors, surveying entire assembly lines at a glance. Ubiquitous digital monitors provided real-time data for all to see. A rainbow palette of hats and uniforms instantly pegged the position and rank of fourteen thousand employees, among whom Bernstein’s undergrads were discreetly embedded.
The students, who’d been born in China, lived, worked, ate, and socialized with their colleagues. They found that, as in any high-pressure workplace, employees were constantly fashioning more agile strategies for getting the job done, often bypassing procedures handed down by bureaucrats from on high. What amazed Bernstein was the lengths to which workers went to conceal their innovations. One told him, “It’s most efficient to hide it now and discuss it later.” Ironically, the plant’s transparency enabled this subterfuge: the colored hats that allowed management to identify each worker’s position also let workers spot managers from across the floor, giving them time to camouflage their improvisations. As the managers made their rounds, they became loci of decreased productivity.
Bernstein’s team hung a curtain in the factory, preparing to conduct an experiment that involved observing assembly lines on one side, and then comparing them with the control lines on the other. But when a worker offhandedly mused about how much better it would be if the curtain completely closed them in on all sides, Bernstein went with her idea, instead. In just a week, the mere shielding of these assembly lines increased productivity by as much as ten to fifteen per cent—a change that persisted for the five months that Bernstein was able to continue observing. Instead of hiding their various tweaks and improvements, the workers behind the curtains could share them with their colleagues without fear of drawing unwanted attention; they even sometimes cross-trained one another, making it easier for people who worked different positions to cover for one another. “When you get watched, you lose autonomy,” Bernstein told me. The result is what he calls the transparency paradox: well-intentioned measures meant to increase transparency and promote productivity, creativity, and compliance often accomplish the opposite. Observation is not a neutral process; human beings react.
Transparency in government, to the extent that it exists, has been hard-won. For most of its history, Congress could be witnessed only by the few members of the public lucky or connected enough to snag seats in the gallery. From time to time, radio broadcasts captured Congress at work; in 1947, a single live-television broadcast of the opening session of the House of Representatives served as the first salvo in the fight for fuller public access. It was followed by spotty televised coverage of the occasional Presidential speech, and then by an outright ban on television cameras in Congressional sessions and committee hearings, in 1952. In the Watergate-chastened nineteen-seventies, calls for transparency led to a brief test of closed-circuit television, in 1977.
That year, a journalist named Brian Lamb submitted a proposal to the governing body of the newly emerging cable-television industry suggesting the creation of a public-affairs channel to show gavel-to-gavel coverage of congressional proceedings. The Cable-Satellite Public Affairs Network (C-SPAN) launched in 1979. It touted itself as “your unfiltered view of government”; the comedian Elayne Boosler has referred to it as “Wild Kingdom but with congressmen.” C-SPAN was hailed as a win for transparency and accountability. Representative Al Gore, who was the very first lawmaker to appear on C-SPAN, argued that it offered “a solution for the lack of confidence in government.” Because it does not filter or edit coverage, C-SPAN is generally considered unbiased—a transparent view.
Along with other lawmakers, Newt Gingrich—who arrived in Congress shortly after C-SPAN did—quickly realized the potential of the medium. Members of Congress delivered more pugnacious speeches at the end of the day, which appeared to stun their opponents into silence—sometimes because nearly everyone had gone home, a fact that the cameras didn’t reveal. (Later, cameras started panning the room.) The researchers Gloria Gennaro and Elliott Ash have examined the so-called C-SPAN effect, demonstrating a correlation between the “emotionality” of lawmakers’ rhetoric and the levels of C-SPAN viewership in their districts. They also found that exposure on C-SPAN has no measurable effect on legislative efforts—perhaps because such efforts often require detailed deal-making and uncomfortable trade-offs that are easier managed in quiet backrooms. C-SPAN, therefore, might not be showing us lawmakers in the wild so much as encouraging them to grandstand to the camera in ways that can be antithetical to the actual work of governing. (By contrast, increased coverage by local news does appear to correlate with legislative progress, though without impact on the emotionality of lawmakers’ oratory.)
These counterintuitive effects sit awkwardly alongside our valorization of transparency. David Pozen, a soft-spoken law professor at Columbia University, told me point-blank that “transparency has a proven track record of failing to deliver on its ostensible benefits.” Open-file discovery, for example, is a transparency effort intended to level the playing field during litigation; it requires prosecutors to share the results of their investigations with defense teams, so that defendants can make more informed decisions about plea bargains. These disclosures also protect defendants from being surprised during trial by witnesses or evidence for which they haven’t prepared. But analyses have found that these laws haven’t always resulted in meaningful improvements for defendants. This might be because most defendants are poor, and their overworked, underpaid public defenders can’t effectively utilize the additional information. But police and prosecutors may also change their behavior once they know that whatever they uncover will be shared with the defense.
Transparency, Pozen told me, “invites conceptual confusion about whether it’s a first-order good that we’re trying to pursue for its own sake, or a second-order good that we’re trying to use instrumentally to achieve other goods.” In the first case, we might feel that transparency is an ideal always worth embracing, whatever the costs. In the second, we might ask ourselves what it’s accomplishing, and how it compares with other routes to the same end.
“There is a standard view that transparency is all good—the more transparency, the better,” the philosopher C. Thi Nguyen, an associate professor at the University of Utah, told me. But “you have a completely different experience of transparency when you are the subject.” In a previous position, Nguyen had been part of a department that had to provide evidence that it was using state funding to satisfactorily educate its students. Philosophers, he told me, would want to describe their students’ growing reflectiveness, curiosity, and “intellectual humility,” but knew that this kind of talk would likely befuddle or bore legislators; they had to focus instead on concrete numbers, such as graduation rates and income after graduation. Nguyen and his colleagues surely want their students to graduate and earn a living wage, but such stats hardly sum up what it means to be a successful philosopher.
In Nguyen’s view, this illustrates a problem with transparency. “In any scheme of transparency in which you have experts being transparent to nonexperts, you’re going to get a significant amount of information loss,” he said. What’s meaningful in a philosophy department can be largely incomprehensible to non-philosophers, so the information must be recast in simplified terms. Furthermore, simplified metrics frequently distort incentives. If graduation rates are the metric by which funding is determined, then a school might do whatever it takes to bolster them. Although some of these efforts might add value to students’ learning, it’s also possible to game the system in ways that are counterproductive to actual education.
Transparency is often portrayed as objective, but, like a camera, it is subject to manipulation even as it appears to be relaying reality. Ida Koivisto, a legal scholar at the University of Helsinki, has studied the trade-offs that flow from who holds that camera. She finds that when an authority—a government agency, a business, a public figure—elects to be transparent, people respond positively, concluding that the willingness to be open reflects integrity, and thus confers legitimacy. Since the authority has initiated this transparency, however, it naturally chooses to be transparent in areas where it looks good. Voluntary transparency sacrifices a degree of truth. On the other hand, when transparency is initiated by outside forces—mandates, audits, investigations—both the good and the bad are revealed. Such involuntary transparency is more truthful, but it often makes its subject appear flawed and dishonest, and so less legitimate. There’s a trade-off, Koivisto concludes, between “legitimacy” and “the ‘naked truth.’ ”