Vic Case Worker Turns to ChatGPT for Drafting Child Protection Report

Victoria's Department of Families, Fairness and Housing (DFFH) has been instructed to impose a ban on generative AI tools following a troubling incident in which a child protection worker utilized ChatGPT to draft a report submitted to the Children’s Court. The investigation led by the state’s information commissioner revealed that the report contained inaccurate personal information and significantly downplayed the risks to the child involved. Although the outcome of the child's case was not affected, the potential for harm was evident, prompting concerns from the Office of the Victorian Information Commissioner (OVIC).

The report, which is supposed to reflect the child protection worker's thorough assessment of the child's risks and needs as well as the parents' capacity to ensure the child's safety and development, was compromised by the use of generative AI. Specifically, ChatGPT was implicated in articulating the dangers a young child faced if they remained in the home with parents charged with sexual offenses. The case worker's decision to input sensitive, case-specific information into ChatGPT constituted a serious violation of state privacy regulations.

According to OVIC, the act of sharing this information with OpenAI, a foreign company, meant that sensitive data was released beyond the control of the DFFH. The report also indicated that OpenAI now possesses this information and retains the authority to determine its future use and dissemination.

In evaluating the report, OVIC identified numerous indicators suggesting the involvement of ChatGPT, including inaccuracies in personal details and inconsistencies with the language and sentence structures expected based on the department’s training and child protection guidelines. Furthermore, it was revealed that the use of ChatGPT may have extended beyond this isolated case. An internal review by the department highlighted 100 instances in the past year where ChatGPT might have been used to draft documents related to child protection. Alarmingly, during the latter half of 2023, nearly 900 employees—almost 13 percent of the department's workforce—accessed the ChatGPT website, indicating a broader trend in generative AI use within the organization.

In the aftermath of the investigation, DFFH acknowledged that there had been an unauthorized use of ChatGPT in drafting the report but contended that there was no concrete evidence suggesting that staff members had used the AI tool to generate content for sensitive matters. This assertion was met with skepticism from OVIC, which firmly disputed the department's claims. OVIC highlighted that the evidence indicated a broader pattern of generative AI usage within the organization, contrary to DFFH's characterization of the incident as a singular occurrence.

The department maintained that the use of generative AI tools was not prevalent, yet the findings from the internal review revealed significant access to such tools among staff, raising concerns about compliance and oversight in handling sensitive information. This divergence in perspectives between DFFH and OVIC underlines the ongoing challenges that organizations face in balancing technological innovation with the imperative of safeguarding privacy and ensuring responsible data handling practices.

In response to these serious breaches, OVIC issued a compliance notice mandating that DFFH prohibit the use of generative AI tools and implement technical measures to block access to them. The department was expected to provide direction to all staff prohibiting the use of public generative AI tools and to establish technical controls to restrict access to a list of specified AI text tools, including ChatGPT, until November 5. Following the incident, the case worker involved is no longer employed by the department, highlighting the severity of the breach and the commitment to uphold the integrity of child protection practices.