LC Labs AI Planning Framework

About

LC Labs has been exploring how to use emerging technologies to expand the use of digital materials since our launch in 2016. We quickly saw machine learning (ML), one branch of artificial intelligence (AI), as a potential way to provide more metadata and connections between collection items and users. Experiments and research have shown the risks and benefits of using AI in libraries, archives and museums (LAMs).

To account for these challenges and realities, LC Labs has been developing a planning framework to support the responsible exploration and potential adoption of AI at the Library. Read the full overview of the Labs framework in our related post on the Signal Blog.

At a high level, the framework includes three planning phases: 1) Understand 2) Experiment and 3) Implement, which supports the evaluation of three elements of ML: 1) Data; 2) Models; and 3) People. We rely on a set of worksheets, questionnaires, and workshops to engage stakeholders and staff and identify priorities for future exploration. The mechanisms, tools, collaborations, and artifacts together form the AI Planning Framework. Our hope in sharing the framework and associated tools in this initial version is to encourage others to try it out and to solicit additional feedback.

In planning for and conducting AI and ML experiments at the Library of Congress, we’ve simplified ML processes into three main elements: Data, Models, and People. The details of all three elements and how they are put together helps us understand whether an application of this technology is useful, ethical and effective.

Elements of Machine Learning / AI

Data

An icon comprised of a black cylinder with three segments

  • Library content
  • Data readiness
  • Training data
  • Tuning data
  • Validation data
  • Target data
  • Output data

Models

An icon comprised of a matrix of four cells, each containing a math symbol

  • End-to-end workflow or pipeline
  • Architectures
  • Type of training
  • Libraries utilized
  • Frameworks or platforms

People

An icon comprised of a four human figures with their arms raised

  • Develop use cases
  • Represented in the data
  • Design and sell AI systems
  • Impacted by AI systems
  • Evaluate and implement

Download this table as an image

Considering the data, models and people involved in an AI system is baked into our AI Planning Framework. The Understand, Experiment and Implement steps include collaborative activities and result in documentation that inform the development of practices and policies for responsible AI. We have yet to move all the way through to the implement stage, but this planning process will build the foundation for a solid and responsible AI strategy based on evidence and Library strengths.

LC Labs AI Planning Phases

Understand

An icon comprised of a lightbulb containing a network of people, institutions, and technology

Experiment

An icon comprised of three arrows and three dots organized in a circle, representing a cycle

Implement

An icon comprised of a pyramid above an institution, a magnifying glass above a graph, people, and a certificate

→ Governance and Policy →

Download this table as an image

Understand

Collaboratively articulate principles; assess risks and benefits, map needs, priorities and expertise; learn about data readiness.

Tools for use in this phase:

Title Description Last Revised Download
Use Case Risk Worksheet This questionnaire is meant to assist staff in assessing the risk profile of an AI use case. The risk level will inform planning for the level of the risk mitigation efforts, estimated timeline for safety, quality and performance verification, and resources required. 2023-11-15 Link to worksheet.
Phase II Risk Analysis Fill out this worksheet to articulate success criteria, measures, risks, and benefits for an AI Use Case. 2023-10-30 Link to worksheet
Data Readiness Assessment Questionnaire to assess readiness and availability of data for the proposed use case. 2023-11-14 Link to questionnaire.

Experiment

We use the following tools and mechanisms for experiments:

The Digital Innovation Indefinite Delivery Indefinite Quantity (IDIQ) contract The Data Processing Plan documents data transformations and the predicted and actual AI model performance for specific tasks. It combines elements from a model card, data cover sheet and documents curatorial provenance. Vendors are required to fill it out as part of the Digital Innovation IDIQ. In Development: NLP vendor evaluation guide and quality review recommendations. Under Recommendation: Balanced datasets for benchmarking newly available AI models and tools.

Test specific use cases, models and data with staff and users to document performance and build quality baselines and benchmarks

Tools for use in this phase:

Title Description Last Revised Download
Data Processing Plan This template documents data transformations and the predicted and actual AI model performance for specific tasks. It combines elements from a model card, data cover sheet and documents curatorial provenance. Vendors are required to fill it out as part of the Digital Innovation IDIQ. 2021-12-01 Attachment J2 on the Library of Congress Digital Innovation IDIQ solicitation
Digital Innovation Indefinite Delivery Indefinite Quantity (IDIQ) The Library of Congress Digital Innovation IDIQ contract is a multi-year contracting mechanism that we can use to fulfill individual AI experiment at the Library of Congress, and includes requirements that may be valuable to the broader community. 2022-07-28 Library of Congress Digital Innovation IDIQ solicitation

Implement

AI or ML services are operational and supported by strategy, policies, integrations, shared quality standards and a skilled workforce.

This phase will include tools to assist with:

Framework Activities

Understand

An icon comprised of a lightbulb containing a network of people, institutions, and technology

Use tools to collaborate and assess:

  • Risks and benefits
  • Principles and values
  • Data readiness
  • Local and domain expertise

Experiment

An icon comprised of three arrows and three dots organized in a circle, representing a cycle

Create practices and documentation to:

  • Test data and models with use case
  • Review output with staff and users
  • Build baselines

Implement

An icon comprised of a pyramid above an institution, a magnifying glass above a graph, people, and a certificate

Create policies and standards, including:

  • Strategy and roadmap
  • Skills and capacities
  • Monitoring and measuring
  • Shared quality standards

Download this table as an image