main_appendix

Stratman, Boutilier, and Albert

Decision-Aware Predictive Model Selection for Workforce Allocation

Decision-Aware Predictive Model Selection for Workforce Allocation

Eric G. Stratman Department of Industrial and Systems Engineering, University of Wisconsin-Madison, egstratman@wisc.edu

Justin J. Boutilier Telfer School of Management, University of Ottawa, boutilier@telfer.uOttawa.ca

Laura A. Albert Department of Industrial and Systems Engineering, University of Wisconsin-Madison, laura@engr.wisc.edu

Many organizations depend on human decision-makers to make subjective decisions, especially in settings where information is scarce. Although workers are often viewed as interchangeable, the specific individual assigned to a task can significantly impact outcomes due to their unique decision-making processes and risk tolerance. In this paper, we introduce a novel framework that utilizes machine learning to predict worker behavior and employs integer optimization to strategically assign workers to tasks. Unlike traditional methods that treat machine learning predictions as static inputs for optimization, in our approach, the optimal predictive model used to represent a worker’s behavior is determined by how that worker is allocated within the optimization process. We present a decision-aware optimization framework that integrates predictive model selection with worker allocation. Collaborating with an auto-insurance provider and using real-world data, we evaluate the effectiveness of our proposed method by applying three different techniques to predict worker behavior. Our findings show the proposed decision-aware framework outperforms traditional methods and offers context-sensitive and data-responsive strategies for workforce management.

Workforce allocation, Predictive modeling, Personalization, Subjective decision-making, Decision-aware learning

1 Introduction

Humans supplement known information with intuition and experience to make decisions. This ability enables them to make informed decisions, even in situations with limited data and uncertain outcomes. As a result, despite the rise of artificial intelligence and automation, many organizations still rely on humans to assess value and risk in decision-making. For example, a mortgage underwriter at a bank decides whether to approve or deny a loan application; a doctor in a medical practice advises a patient on undergoing a procedure; and a hiring manager in a corporation determines whether to hire a candidate. These decisions inherently rely on a worker’s subjective judgment, which complements objective information.

Although uniform decision-making within a workforce might seem ideal, a worker’s personal judgment and experience invariably shape their decision-making process. For example, one poor decision may alter a worker’s behavior when faced with a similar decision in the future. Similarly, psychological studies have shown that personality influences risk aversion, making outcomes highly dependent on the worker assigned to a particular decision (Becker et al. 2012, Lönnqvist et al. 2015, El Othman et al. 2020). Recognizing this, aligning task assignments with a worker’s unique characteristics and experience can enhance both the effectiveness and efficiency of an organization’s decision-making processes.

In this paper, we explore worker individuality in subjective decisions and investigate leveraging worker tendencies in a workforce allocation problem. We aim to uncover how unique worker tendencies can be effectively utilized in the assignment of tasks, focusing on aligning tasks with workers most likely to make beneficial decisions. Existing research that studies worker allocation typically assumes that worker behavior is uniform or that worker attributes are known with certainty (De Bruecker et al. 2015). However, in many real-world applications, behavior varies between individuals and must be learned from data. We examine the challenge of learning worker behavior using real-world data and explore how different approaches impact the resulting workforce allocation problem.

The framework we present leverages machine learning to model worker behavior in decision-making tasks in combination with integer optimization to assign workers to tasks. The goal is to improve decision-making outcomes by aligning task assignments with each worker’s unique characteristics, while supporting the organization’s broader objectives. Our framework is not intended to replace human decision-makers. Instead, our approach aims to complement human judgment by aligning task assignments with workers’ unique traits, thus supporting the organization’s broader objectives without removing individual decision-making nuances. For example, if an organization wishes to reduce risk, the optimization model may assign a high-risk task to a worker who has demonstrated risk aversion to similar tasks in the past. Worker assignments are therefore highly dependent on the insights gained from the predictive model.

The predictive models used to represent worker behavior and their connection to the optimization task are an important consideration. Several approaches exist for incorporating predictive models into optimization tasks, each varying in complexity and effectiveness. The simpliest approach – the Decoupled Prediction Optimization (DPO) method – treats prediction and optimization separately. Predictions are made as single-point estimates and then used as deterministic parameters within the optimization model. While DPO’s simplicity is appealing, the major drawback is that it does not consider the accuracy of the predictions, potentially leading to less robust decisions.

Uncertainty-Aware Optimization (UAO) takes a more nuanced approach by accounting for prediction error or parameter uncertainty within the optimization model. UAO typically employs robust or stochastic optimization methods to account for uncertainty. This approach aims to generate solutions that are resilient to the uncertainty inherent in predictive models, thus enhancing decision quality. However, uncertainty may not be uniform and prediction errors on tasks more critical to the optimization model tend to have a greater impact on generated solutions. Given the separation between the prediction and optimization phases, the prediction model cannot directly account for this interplay, potentially overlooking opportunities to fully harness predictive insights for enhancing optimization decisions.

Decision-aware prediction optimization (DAO) aims to overcome the separation of prediction and optimization by adapting the predictive modeling process to align closely with the specific optimization objective. DAO focuses on tailoring the predictive task to minimize decision loss, defined as the negative impact of prediction inaccuracies on subsequent optimization efforts. Research in this field, such as Elmachtoub and Grigas (2022), Kallus and Mao (2023) and Chung et al. (2022), has primarily investigated how the loss functions of machine learning models can be adjusted to minimize the downstream decision loss. However, loss function formulation is just one element of machine learning. Decision loss and the effectiveness of optimization decisions can also be significantly impacted by other critical aspects of the machine learning to optimization pipeline, such as feature engineering, the choice of modeling algorithms, and the strategies employed for data aggregation and preparation. These considerations can even be significantly more important than the loss function in the learning process, particularly in scenarios characterized by complex prediction tasks and limited data.

To illustrate this, consider the task of learning individual worker behavior within a workforce. To learn each individual’s unique behavior, one could train an individualized predictive model for each worker built using only that worker’s data. These individualized models reflect each worker’s unique decision-making history; however, as is common for human-centric processes, there may not be enough data to build a sufficiently complex model that represents the behavior of each worker. Alternatively, one could pool data across individuals to create an aggregate predictive model. This aggregated model benefits from a richer dataset, allowing for the application of more sophisticated modeling techniques and potentially enhancing overall performance. However, this approach overlooks the distinct nuances of individual workers’ behaviors. This dilemma highlights a balance between achieving broad model performance and maintaining a degree of personalization. It is unclear if the choice that minimizes prediction loss and model accuracy aligns with minimizing decision loss within the downstream optimization problem.

In this paper, we propose and examine a framework that extends DAO to account for predictive model selection. This enables an exploration of diverse predictive modeling methodologies, moving beyond the traditional focus on loss function formulation. Our framework considers a workforce comprised of individual workers, each with unique decision-making characteristics, and optimally selects a predictive model for each worker to be used in a downstream optimization problem. By integrating model selection into the DAO, we bridge the gap between individual worker tendencies and overarching organizational goals, enhancing the efficiency and effectiveness of decision-making processes. This project, conducted in collaboration with a U.S.-based auto-insurance provider, uses real-world data to demonstrate our framework’s practical potential. The insurer employs underwriters to evaluate high-risk policy requests, with tasks currently assigned randomly across a large workforce. The insurer’s leadership suspects that underwriters’ decisions vary based on personal experience and risk tolerance. By modeling these individual tendencies using machine learning, we assess their impact within a downstream optimization framework, demonstrating that targeted predictive model selection can refine workforce allocation and create additional value for the organization.

The key contributions of this work, driven by the need for decision-aware model selection in real-world settings, are summarized as follows:

1. Decision-Aware Workforce Allocation Framework: We propose a novel optimization framework that directly integrates predictive model selection into the workforce allocation process. Unlike traditional methods that treat prediction and optimization as separate steps, our DAO framework enhances the quality of worker-task assignments by dynamically selecting the most appropriate predictive model for each worker within the optimization process. This integration allows for a more accurate alignment of worker behavior with organizational objectives, resulting in improved decision-making and task allocation.

2. Predictive Modeling Approaches for Workforce Allocation: We conduct an evaluation of three distinct predictive models to represent worker behavior in the context of workforce allocation. This analysis highlights the trade-offs between capturing the unique decision-making processes of individual workers and addressing the limitations imposed by data availability. By applying these models to real-world data, we demonstrate their effectiveness in optimizing worker-task assignments and explore their performance within both traditional UAO and our proposed DAO frameworks.

3. Application to Insurance Workforce Management: In collaboration with a leading U.S. auto-insurance provider, we apply our framework to real operational data and conduct an analysis of the results. This analysis includes evaluating the effectiveness of our approach in optimizing workforce allocation and discussing the managerial insights gained from these findings. Our study validates the practical applicability of our methods and provides actionable strategies for managers to enhance workforce management in real-world settings.

The paper begins with a comprehensive literature review in Section 2. Section 3 introduces the workforce allocation problem and discusses methods for incorporating worker behavior into worker-task assignments, starting with traditional DPO and UAO approaches and then our proposed DAO framework. Section 4 presents a case study conducted with a U.S. auto-insurance provider to demonstrate the application of the DAO framework, covering the business context, predictive modeling approaches, and the implementation of UAO and DAO frameworks. Section 5 presents the findings of the case study and discusses key managerial insights. Finally, concluding remarks are provided in Section 6.

2 Literature Review

Our research is related to four streams of literature: workforce allocation, predicting worker behavior, decision-aware optimization frameworks, and operations research applications in the insurance industry.

2.1 Workforce Allocation

Assigning workers to tasks is a well-studied problem within operations research and has applications in many fields including production (Eiselt and Marianov 2008, Vilà and Pereira 2014), healthcare (Lanzarone and Matta 2014), and product management (Hendriks et al. 1999). We refer the reader to the review papers by Van Den Bergh et al. (2013) and Bouajaja and Dridi (2017) for a comprehensive overview.

Many authors study assignment problems while considering worker attributes such as preferences (Dowsland 1998, Akbari et al. 2013), productivity (Brusco and Johns 1998, Brusco 2008), skills (De Bruecker et al. 2015), and performance (Derman et al. 1972, Nikolaev and Jacobson 2010). Most similar to our work are those that individualize these attributes for the specific worker-task assignment. For example, Dowsland (1998) consider the preferences of individual nurses for specific shifts, Hanne et al. (2009) explores company preference for worker-task assignments, and Brusco (2008) leverages departmental productivity of workers. However, these papers often assume worker attributes are known with certainty. Our work contributes to this stream of research by using data-driven methods to derive and understand worker attributes in subjective decision-making tasks, addressing the gap in the literature where uncertainty in worker attributes has been previously identified but not extensively explored (De Bruecker et al. 2015). Furthermore, we explore the impact of uncertainty in these predictions within a downstream optimization problem.

2.2 Predicting Worker Behavior

The application of machine-learning techniques to understand and predict worker behavior is a growing field in operations research and human resource management. This interdisciplinary approach has gained momentum as organizations increasingly recognize the value of data-driven insights into workforce dynamics. Existing research includes predicting workforce attrition (Alduayj and Rajpoot 2018, Alsheref et al. 2022, Raza et al. 2022), productivity (Florez and Cortissoz 2017, Alaloul et al. 2022), and performance (Chalfin et al. 2016, Sajjadiani et al. 2019). These papers typically focus on the methods used to identify worker attributes and the insights learned. Our paper moves beyond prediction and focuses on how the predictions can be used to improve decision processes within an organization. Most similar to our work is the paper by Chalfin et al. (2016) that discusses how individuals within a heterogeneous workforce should be hired or selected for promotions. However, unlike this work, we focus on how the method used to identify worker attributes impacts the value of the generated solution and resulting business decisions.

2.3 Decision-Aware Learning

Recent advancements in research are increasingly highlighting the importance of integrating predictive modeling with decision-making processes. This has led to the development of decision-aware approaches, which are specifically designed to align machine learning algorithms with the unique requirements of optimization problems. Notable contributions in this area include the work of Elmachtoub and Grigas (2022) and Kallus and Mao (2023), who have incorporated decision-loss functions into their models and adjusted decision forest algorithms based on their optimization effects, respectively. However, these approaches often encounter limitations in terms of their broader applicability, since they are heavily dependent on the specific structures of the optimization problems they address. To extend the reach of these methodologies, Chung et al. (2022) proposed a more flexible, model-agnostic approach, utilizing sample weighting derived from mathematical principles to increase their compatibility with a variety of machine learning models. Our study aligns with the approach of Chung et al. in maintaining the conventional structure of machine-learning training algorithms. However, we differentiate our research from these approaches by focusing on the selection of predictive models instead of modifying the training algorithms themselves. Similarly, while Wilder et al. (2019) have explored similar decision-aware approaches in the realm of combinatorial optimization, their emphasis on machine-learning training algorithms contrasts with our approach of predictive model selection. Our research contributes to the field of decision-aware methodologies by advocating for a more comprehensive strategy that not only considers the training of predictive models but also emphasizes the importance of strategic model selection to optimize decision-making processes.

2.4 Operations Research in the Insurance Industry

Lastly, this work contributes to the stream of research that applies operations research methods in the insurance industry. Contributions in this field have focused on using techniques such as game theory, dynamic programming, and stochastic models to address challenges related to underwriting, risk management, and claims processing (Brockett and Xiaohua 1997, Dedu and Şerban 2015, Consigli et al. 2011, Wüthrich 2018). In particular, recent advancements have increasingly incorporated machine learning techniques to improve predictive accuracy and operational efficiency in insurance, focusing on pricing optimization, fraud detection, and risk classification (Blier-Wong et al. 2020, Santos et al. 2024). Our work aligns most closely with the study by Bertsimas and Orfanoudaki (2022), who explore how machine-learning classification thresholds impact risk management in algorithmic insurance. However, rather than treating classification thresholds as parameters we can easily control, we focus on accounting for the human aspect of decision-making, particularly the variability in human behavior and judgment, as it influences the predictive modeling and optimization of worker-task assignments in insurance underwriting.

3 Formulation

In this section, we introduce the workforce allocation problem and outline methods for leveraging worker behavior when making worker-task assignments. In Section 3.1, we begin by presenting a general workforce allocation problem and discuss traditional approaches for incorporating predictive insights into worker-task assignments. Although these methods are straightforward, they treat prediction and optimization as separate steps, neglecting how the predictive model used to represent worker behavior influences the resulting assignment policy. To address this disconnect and improve solution quality, in Section 3.2, we propose a novel decision-aware workforce allocation framework. This method enhances worker assignments by directly integrating predictive model selection for worker behavior into the optimization framework.

3.1 Workforce Allocation Problem

We first introduce the general workforce allocation problem. In this problem, a workforce of various workers is allocated to various tasks. Let $I$ denote the set of workers and $J$ the set of tasks. We define the decision matrix $X\in\{0,1\}^{|I|\times|J|}$, where $x_{ij}=1$ if worker $i$ is assigned to task $j$, and $x_{ij}=0$ otherwise. For now, we keep the remaining feasible region of assignments, $\mathcal{X}$, general but note that worker assignments are typically constrained by factors such as worker capacity and skill prerequisites.

Once a worker is assigned tasks, each task requires a binary decision from the assigned worker. For example, such tasks might include deciding whether to accept or reject an auto insurance request, prescribing medication to a patient, or making a hiring decision for a job candidate. We introduce the Bernoulli random variable $B_{ij}\in\{0,1\}$ to represent the behavior of worker $i$ when assigned to task $j$. In particular, $B_{ij}=1$ indicates that worker $i$ takes an action (e.g., approving a request) on task $j$, while $B_{ij}=0$ indicates that an action is not taken. The value of performing an action for task $j$ is denoted by $v_{j}\in\mathbb{R}$, where $v_{j}$ can be positive or negative, depending on the associated risks or benefits of the task. Our objective is to maximize the total expected value of the actions taken, subject to the constraints defined by the feasible region $\mathcal{X}$. Formally, the workforce allocation problem is defined as:

maximize $$\sum_{i\in I}\sum_{j\in J}\mathbb{E}\left[B_{ij}\right]v_{j}x_{ij}$$ (1a)

subject to $$\ x_{ij}\in\{0,1\},\quad\forall i\in I,j\in J,$$ (1b)

$$X\in\mathcal{X}.$$ (1c)

The primary challenge in this optimization model is the treatment of the expected value in the objective. One naive approach is to assume we have knowledge of worker behavior and replace $E[B_{ij}]$ with a deterministic parameter. In reality, due to the subjective nature of tasks, $B_{ij}$ is often uncertain until worker behavior is observed. To inform worker behavior, a reasonable approach is to utilize a predictive modeling technique such as a machine-learning model. These predictive models can utilize historical data to predict worker actions, which can be used in optimizing worker allocation.

3.1.1 Decoupled Prediction Optimization (DPO).

The simplest method to integrate predictive modeling into an optimization framework is the DPO approach. In this method, a trained predictive model $m$ is used to directly predict the action of worker $i$ on task $j$. Specifically, let $B^{m}_{ij}\in\{0,1\}$ denote the predicted action made by model $m$ for worker $i$ on task $j$. The DPO formulation is as follows:

maximize $$\ \sum_{i\in I}\sum_{j\in J}B^{m}_{ij}v_{j}x_{ij}$$ (2a)

subject to $$\ x_{ij}\in\{0,1\},\quad\forall i\in I,j\in J,$$ (2b)

$$X\in\mathcal{X}.$$ (2c)

Although the decision maker typically selects a predictive model that maximizes prediction accuracy, a shortcoming of this approach is the assumption that these predictions accurately represent $B_{ij}$. In practice, it can be difficult to assess if the optimization model is exploiting the true behavior of each worker or potential inaccuracies of the predictive models. If the latter, then the optimization model may produce solutions that perform poorly in practice.

3.1.2 Uncertainty-Aware Optimization (UAO).

While the DPO approach offers a straightforward framework for integrating predictive modeling with optimization tasks, it assumes that the predictions are equally accurate. To mitigate this assumption, UAO is based on the idea that $\mathbb{P}(B_{ij}^{m}=1)\approx\mathbb{E}[B_{ij}]$. This representation intuitively captures the models confidence that $B_{ij}=1$, which may vary across $i-j$ pairs. The UAO formulation can be written as:

maximize $$\ \sum_{i\in I}\sum_{j\in J}\mathbb{P}(B^{m}_{ij}=1)v_{j}x_{ij}$$ (3a)

subject to $$x_{ij}\in\{0,1\},\quad\forall i\in I,j\in J,$$ (3b)

$$X\in\mathcal{X}.$$ (3c)

Although the UAO approach captures predictive model confidence, $\mathbb{P}(B^{m}_{ij}=1)$ can vary depending the choice of predictive model (e.g., decision tree versus logistic regression), data preparation techniques, and training process. In particular, for UAO, predictive model selection occurs independently from the optimization process.

3.2 Decision-Aware Optimization (DAO)

For many workforce settings, predicting human behavior is challenging due to the complexity of decision-making processes and the limited data typically generated from manual human activities. Traditional machine learning approaches gauge the effectiveness of predictive modeling using metrics such as accuracy or area-under-the-curve (AUC). However, these metrics may not fully capture the practical implications of prediction errors in operational contexts.

Our proposed DAO framework addresses this challenge by embedding predictive model selection directly within the optimization process. Unlike DPO and UAO approaches where the outputs of predictive models are treated as input to the optimization model, DAO acknowledges that predictions influence the decisions made within the optimization framework. Our framework aims to adapt predictive model selection to better support decision making, thereby enhancing the overall effectiveness and reliability of the optimization model solution.

To implement the DAO approach, we consider a set of candidate predictive models, denoted by $M(i)$, for predicting each worker’s behavior. Each model $m\in M(i)$ estimates the behavior of worker $i$ on task $j$, represented by $B_{ij}^{m}\in\{0,1\}$. We introduce binary decision variables $y_{im}$ to indicate the predictive model selection, where $y_{im}=1$ indicates that model $m$ is selected to represent worker $i$ and $y_{im}=0$ if predictive model $m$ is not selected for worker $i$. Additionally, we introduce binary variables $z_{ijm}$ to link the work allocation with predictive model selection. The variable $z_{ijm}=1$ indicates that worker $i$ is allocated to task $j$ using the predictions from model $m$ and $z_{ijm}=0$ otherwise. The DAO formulation can be written as:

maximize $$\ \sum_{i\in I}\sum_{j\in J}\sum_{m\in M(i)}\mathbb{P}(B^{m}_{ij}=1)v_{j}z_{ijm}$$ (4a)

subject to $$\sum_{m\in M(i)}z_{ijm}=x_{ij},\quad\forall i\in I,j\in J,$$ (4b)

$$z_{ijm}\leq y_{im},\quad\forall i\in I,j\in J,m\in M(i)$$ (4c)

$$\sum_{m\in M(i)}y_{im}=1,\quad\forall i\in I,j\in J,$$ (4d)

$$z_{ijm}\in\{0,1\},\quad\forall i\in I,j\in J,m\in M(i),$$ (4e)

$$y_{im}\in\{0,1\},\quad\forall i\in I,m\in M(i),$$ (4f)

$$x_{ij}\in\{0,1\},\quad\forall i\in I,j\in J,$$ (4g)

$$X\in\mathcal{X}.$$ (4h)

The objective function (4a) aims to maximize the expected value of workforce allocations by simultaneously optimizing predictive model selection and worker-task assignments. Constraint sets (4b) and (4c) ensure each worker-task assignment is represented by one of the selected predictive models. Constraint set (4d) ensures that each worker is assigned exactly one predictive model from their set $M(i)$. Constraint sets (4e)-(4g) enforce the binary requirements on the decision variables. Constraint (4h) enforces the worker allocation requirements represented by the set $\mathcal{X}$.

We note that the DAO formulation generalizes both the UAO and DPO approaches. The DAO formulation simplifies to a UAO model when $|M(i)|=1\forall i\in I$. In this case, the predictive model used to represent each worker is predetermined (i.e., $y_{im}=1$) and $x_{ij}=z_{ijm}$. The DAO formulation simplifies to the DPO approach when $|M(i)|=1$ and the $B_{ij}^{m}$ are assumed to be deterministic (i.e.,$\mathbb{P}(B^{m}_{ij}=1)=B^{m}_{ij}$). Lastly, we note that the constraint set (4d) serves as a form of regularization because it restricts the selection of one predictive model per worker, which helps to prevent overfitting. While relaxing this constraint could introduce more flexibility in model selection, we retain it to ensure simplicity, interpretability, and to reduce the risk of overfitting by avoiding overly complex model choices for individual workers.

3.2.1 Adjusting Model Predictions.

A challenge when assigning tasks to workers using multiple predictive models is that the accuracy of each model $m\in M(i)$ can vary significantly depending on the task. For example, if a predictive model $m\in M(i)$ is trained using data from multiple workers, it may perform well for tasks where there is a high level of consensus among workers. However, for tasks where decision-making varies significantly across workers, the same model might perform poorly, as it may fail to capture the subtleties of individual worker behavior in more subjective scenarios.

Furthermore, in subjective decision-making contexts, tasks are often unique, making it difficult for predictive models trained on historical data to generalize effectively to new tasks. For instance, a random forest model might estimate the probability $\mathbb{P}(B_{ij}^{m}=1)$ as the proportion of decision trees that predict $B_{ij}=1$, based on past worker behavior. However, given the diverse nature of tasks that require nuanced human judgment, new tasks may introduce variations not previously captured in the training data. In such settings, relying solely on historical training data can lead to predictions biased by the observed training data and may not accurately reflect the true dynamics of current tasks.

To mitigate this issue, we propose adjusting the expected behavior prediction $\mathbb{E}[B_{ij}]$ by incorporating the predictive model’s performance on a test dataset. This adjustment helps account for the model’s ability to generalize beyond the training data and ensures a more accurate representation of worker behavior for future tasks.

We begin by expressing the expected behavior $\mathbb{E}[B_{ij}]$ as follows:

$$\mathbb{E}[B_{ij}] =\mathbb{P}(B_{ij}^{m}=1),$$ (5a)

$$=\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=1)\cdot\mathbb{P}(B_{ij}^{m}=1)$$ (5b)

$$\quad+\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=0)\cdot\mathbb{P}(B_{ij}^{m}=0).$$

In this equation $\mathbb{P}(B_{ij}^{m}=1)$ and $\mathbb{P}(B_{ij}^{m}=0)$ represent the likelihood that the predictive model $m$ forecasts a specific action by worker $i$ on task $j$. The terms $\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=1)$ and $\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=0)$ capture the conditional probabilities that worker $i$ actually takes the action, given the model’s prediction. Since these conditional probabilities $\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=1)$ and $\mathbb{P}(B_{ij}=1\mid B_{ij}^{m}=0)$ cannot be directly observed until worker behavior is observed, we estimate these terms using the model’s performance on a test dataset. This allows us to better gauge how well the model generalizes to new tasks. Thus, we adjust the expected behavior prediction as follows:

$$\mathbb{E}[B_{ij}] \approx\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=1)\cdot\mathbb{P}(B_{ij}^{m}=1)$$ (6)

$$\quad+\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=0)\cdot\mathbb{P}(B_{ij}^{m}=0).$$

Note that $\mathbb{P}(B_{ij}^{m}=1)$ and $\mathbb{P}(B_{ij}^{m}=0)$ are known from the predictive model, while the terms $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=1)$ and $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=0)$ represent the estimated conditional probabilities of the worker’s actual behavior, derived from the test dataset’s results. Several techniques can be employed to estimate these conditional probabilities, including clustering methods and simple predictive models. Furthermore, when the prediction error is uniformly distributed, $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=1)$ and $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=0)$ correspond to the positive predictive value and the false omission rate of model $m\in M(i)$ when applied to worker $i$ (Parikh et al. 2008).

By adjusting model predictions based on test dataset performance, we integrate out-of-sample predictive performance into our decision-making process. This method enhances both the accuracy and relevance of task assignments. Additionally, employing estimated conditional probabilities offers a richer analysis than straightforward accuracy metrics alone. This approach aligns model predictions with individual decision-making tendencies, including factors like risk aversion and tolerance, ensuring that assignments reflect true worker behavior more closely.

4 Case Study: Insurance Underwriter Allocation

To demonstrate the application of the proposed DAO framework, we partnered with a leading U.S.-based auto-insurance provider that manages a workforce of insurance underwriters who are responsible for reviewing and making decisions on unusual or high-risk insurance policy requests. The insurance industry offers a high-stakes financial environment, making it an ideal setting for exploring workforce allocation challenges.

In this section, we present the details of this case study. First, in Section 4.1, we provide the motivation and details of the business context. Next, in Section 4.2, we present the real-world dataset and propose three predictive modeling approaches for learning underwriter behavior. In Section 4.3, we introduce the UAO and DAO models used to allocate workers to tasks. The results of the case study, along with key managerial insights, are presented in Section 5.

4.1 Business Context

In this partnership, we examine the allocation of insurance underwriters. Underwriters are employees that evaluate the details and terms of high-risk or unusual insurance policy requests and determine whether the company should approve the requested coverage. This decision-making process involves a binary choice, either to accept or reject the policy. High-risk policies have the potential to be particularly profitable for the insurance company, as customers typically pay a higher premium (i.e., the payment made by the customer for insurance coverage). However, these policies also come with a greater risk of financial loss due to the inherent uncertainty and unpredictability associated with insuring high-risk individuals or situations.

Traditionally, the task of reviewing a particular high-risk policy is assigned to an underwriter randomly. Although underwriters undergo standardized training, we hypothesize that individual differences, such as varying levels of experience and risk tolerance, may lead to variations in their performance. Our objective is to identify these tendencies and demonstrate how incorporating such insights into the workforce allocation process can support an organizational goal, without creating additional work for the employees. For example, if the insurance company believes a policy should be accepted, it may wish to avoid assigning it to an underwriter who has historically shown aversion to similar policies.

To inform worker allocation, we need to predict how underwriters will behave on future policies, which is challenging for two key reasons. First, each underwriter deals with a wide variety of unique policy requests, each with distinct characteristics. This variability makes it difficult to generalize patterns across policy requests as the decision criteria for one type of policy may not apply to another. Second, because underwriting is a manual and judgment-based process, each underwriter generates only a limited amount of data that captures their decision-making patterns. This scarcity of data further complicates predictive modeling as the available data may not fully reflect the nuances of each underwriter’s approach to different policy types. These limitations underscore the need for the strategic application of predictive modeling insights. This is where the DAO method proves valuable, enabling more tailored and effective assignments that align with underwriter strengths.

4.2 Learning Underwriter Behavior

In learning underwriter behavior, our goal is to predict whether a particular underwriter will accept or reject a policy given the details of the policy request. The dataset for this case study was provided by the auto-insurance company and includes deidentified requests for auto insurance coverage along with the decisions made by the assigned underwriters. The policy details include relevant information such as reasons for underwriter review, customer demographics (age and years driving), vehicle specifications, intended vehicle use, and requested coverage. After removing correlated features (above $0.85$ or below $-0.85$) and before converting categorical variables to binary features (i.e., one-hot encoding), the dataset contains 56 features in total.

We use 15 months of data to develop our predictive models. Twelve months of data were used for training the predictive models, and data from the subsequent 3 months were used for testing. A final month of data is reserved for the optimization phase of the case study. Underwriters that reviewed fewer than 50 policies in the testing dataset were removed and any policy reviewed by more than one underwriter was excluded. After applying these filters, we were left with data for 53 underwriters; 17,853 rows for training and 3,857 rows for testing. The data inclusion process is outlined in Appendix A.

Using this data set, we develop three approaches to predict if an underwriter approves a given policy based on the features of that policy. The approaches include constructing individualized models for each underwriter (Section 4.2.1), developing an aggregate model that leverages all available data (Section 4.2.2), and employing a profile-based approach that combines elements of both individual and aggregate modeling (Section 4.2.3). This range of strategies provides the flexibility to select a predictive model that best aligns with each underwriter’s unique behavior and the data available.

4.2.1 Individual Predictive Model.

We first consider individualized predictive models. For this approach, we develop a unique predictive model for each worker, trained on data generated by that worker’s previous decisions. While this method captures each worker’s unique decision-making processes and behavioral tendencies, it can be limited by the availability of sufficient data, an issue commonly encountered with new hires or in rapidly evolving industries.

We use a logistic regression model with L2 (ridge) regularization for each worker. Logistic regression is selected because it performs well in settings with limited training data and L2 regularization provides robustness against outliers in data sets with many features, particularly in data-scarce settings. The weight of the regularization term is tuned using 10-fold cross-validation on the training set. We use area-under the receiver operating characteristic curve (AUC) as our primary metric.

Figure 1 shows the performance of each individual predictive model quantified by test set AUC. An AUC of 0.5 is equivalent to guessing and larger values indicate higher predictive power. The weighted (by number of observations) average AUC across all individual models was 0.62. Notably, only 11.3-percent of the models achieved an AUC above 0.70, suggesting that the majority of individual predictive models perform only slightly better than randomly guessing a worker’s decisions (Hanley and McNeil 1982, Hosmer Jr et al. 2013).

The low AUC values of the individual predictive models indicate that while these models can provide personalized insights, their overall predictive performance is modest. We hypothesize the low AUC values are primarily due to the limited training data available per worker. In the workforce allocation context, these results highlight an important trade-off. While individual models created using personalized data may capture personalized worker behavior, the effectiveness of these models can be limited by data scarcity. Therefore, in data-limited settings, relying solely on individual predictive models may not yield strong performance. This finding highlights the need to incorporate other predictive modeling strategies, such as aggregate or profile-based models, to improve predictive accuracy.

4.2.2 Aggregate Predictive Model.

To address data scarcity issues that may impact individual predictive model performance, we propose an aggregate predictive model. In this approach, a single predictive model is trained using the available data from all workers with the same features as the individual predictive models. For tasks where there is a consensus among workers, this approach is reasonable as it provides valuable insights into general decision-making patterns. However, a potential limitation is its lack of sensitivity to individual differences, leading to inaccuracies in settings where consensus is low.

Unlike in the individual predictive model, where we are constrained by the need for models that perform well with smaller datasets, the aggregated data allows us to train a random forest model, capable of capturing complex non-linear relationships due to the larger volume of data available. The minimum number of samples in each leaf of the random forest model is tuned using 10-fold cross-validation on the training set.

Figure 1 displays the AUC values obtained from applying the aggregate model to each individual underwriter in the test set. The majority of points lie above the 45-degree line, indicating that the aggregate predictive model achieves a better test set AUC value than the individual predictive model for most underwriters. In fact, the weighted (by number of observations) average AUC of the aggregate predictive model across all individual underwriters was 0.72; an increase of 16-percent over the individual models. Additionally, when applied to specific workers, the aggregate model achieved an AUC of 0.70 or higher for 62.2-percent of the workers, suggesting that it provides moderate to good predictive power for the majority of underwriters (Hanley and McNeil 1982, Hosmer Jr et al. 2013).

Figure 1: Comparison of the test set AUC between individual and aggregate predictive models for each underwriter, with the grey dashed line indicating where the individual and aggregate models achieve equal AUC values.

To further evaluate the aggregate model’s fit for each worker, we compare each worker’s actual behavior to the aggregate model’s predictions. Using the test dataset, we analyze the Positive Predictive Value (PPV) and False Omission Rate (FOR) of the aggregate model for each worker. The results are summarized for 15 of the 53 underwriters with the most training data in Figure 2. A high PPV and low FOR (wide bar) indicate a strong fit. When both metrics are high, the underwriter is more risk-tolerant (bar skewed right), whereas when both are low (bar skewed left), the underwriter is more risk-averse. As shown in Figure 2, workers vary in how well they are represented by the aggregate model and in how they behave with respect to it. This suggests that while the aggregate model may work well for some workers, it may not be suitable for others, further motivating the DAO approach.

Figure 2: Out-of-sample Positive Predictive Value (PPV) and False Omission Rate (FOR) of the aggregate predictive model when applied to fifteen underwriters.

While computing PPV and FOR assesses individual performance against the aggregate model, it does not capture individual tendencies that might be discerned through the more personalized approach of the individual predictive model. The aggregate model is designed to reflect broader trends and commonalities rather than the unique subtleties that might be evident in individual-specific models. Therefore, for worker-allocation, using an aggregate predictive model is more effective in settings where collective behavior outweighs individual differences. However, in environments where decision-making varies significantly across workers, a more personalized predictive modeling approach may be required.

4.2.3 Profile Predictive Model.

To balance the trade-offs between individualized and aggregate predictive models, we also examine a profile-based predictive model. In this approach, workers are clustered according to distinct behavioral patterns, and a predictive model is built for each cluster. This method aims to mitigate the data limitations inherent in individual models while providing more nuanced predictions than the aggregate model.

We clustered underwriters by assessing their actual behavior relative to the aggregate predictive model presented in Section 4.2.2. This categorization was achieved by calculating the true positive and true negative rates within the training sample. Subsequently, k-means clustering was applied to group workers based on these performance metrics. The results of this analysis are displayed in Figure 3, where the true positive and true negative rates for each underwriter are plotted on the x-axis and y-axis, respectively. The clustering process resulted in six distinct clusters, as shown by the shapes and coloring in the figure. The decision to use six clusters was based on the distribution of workers and a thorough analysis of inter-cluster distances, which is further detailed in Appendix B. These clusters represent distinct behavioral groups within the workforce. The data generated by the underwriters in each group is then used to develop a profile predictive model. We employed logistic regression models with L2 regularization for each cluster, similar to the individual predictive models, to avoid issues that may arise from limited data.

Figure 3: In-sample true positive and true negative rates are used to cluster workers for the creation of profile predictive models.

The weighted (by number of observations) average AUC of the profile predictive models across all individual underwriters was 0.67; higher than the individual models and lower than the aggregate. When the profile models were applied to the specific underwriters whose data was used to generate them, 16.7-percent of the underwriters’ models had an AUC below 0.70.

4.2.4 Discussion.

Depending on the specific context and the data at hand, each predictive modeling approach has its unique advantages. Individual models are particularly effective for in-depth analysis of a single worker’s behavior, although their performance can be hindered by a scarcity of data. The aggregate model is better at capturing complex patterns consistent across workers, but may not adequately capture the subtle nuances of each individual’s decision-making process. The profile predictive model blends these two approaches, integrating the detailed insights of individual behaviors with the comprehensive trends observed in aggregate data.

The effectiveness and comparative performance of these models can be visualized in Figure 4, which presents a receiver operating characteristic (ROC) curve and AUC values, illustrating the predictive power of each model when applied to a test data set. As shown in Figure 4, the aggregate predictive model has the highest AUC value of 0.72, indicating higher predictive power. The individual model shows the least predictive power with an AUC of 0.62, while the profile model is in-between with an AUC score of 0.67.

Figure 4: ROC curves and AUC values illustrating the predictive power of the proposed predictive models.

This analysis can be expanded to explore other predictive models, such as different worker groupings or more complex modeling techniques, to potentially enhance prediction accuracy further. The selected predictive models are meant to highlight a range of options that can be used in this type of setting. The choice of predictive model candidates ultimately depends on the specific goals of the analysis, the data available, and the behaviors studied.

In traditional DPO and UAO, we would typically select the predictive model with the highest AUC or apply the same approach uniformly across all workers. However, it is important to recognize that the predictive model with the highest predictive power, as indicated by AUC values, does not always guarantee the best decision performance. In the workforce allocation optimization model, the utility of each predictive model depends not only on its standalone accuracy but also on how each worker is utilized within the workforce allocation framework. In the next section, we explore this idea by applying these methods to the workforce allocation problem, using both the UAO and DAO approaches.

4.3 Optimization Model

We apply these predictive models to the proposed workforce allocation frameworks to demonstrate importance of predictive model selection. Initially, we assume that each worker’s predictive model is predetermined and apply the UAO framework as outlined in Section 3.1.2. This involves implementing the UAO framework using each of the three predictive modeling methods (individual, aggregate, and profile predictive models). Then, we use the DAO framework and optimize predictive model selection within the assignment model itself, as detailed in Section 3.2. We evaluate the effectiveness of the optimized assignment strategies using the one month of data that was set aside from the training and testing datasets.

For the one-month planning horizon, the set $I$ represents the underwriters in the workforce, with $|I|=53$. The set $J$ includes the policies that need to be reviewed, with $|J|=1,187$. The set $M(i)$ contains the predictive models that represent the behavior of underwriter $i\in I$. In the UAO setting, $|M(i)|=1$, meaning each underwriter has one predetermined predictive model. In the DAO setting, $|M(i)|=3$, representing the choice among the three predictive models.

We compute the probability terms $\mathbb{E}[B_{ij}]$ using the approximation presented in equation (6). We estimate $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=1)$ and $\hat{\mathbb{P}}(B_{ij}=1\mid B_{ij}^{m}=0)$ as the PPV and FOR of model $m\in M(i)$, when applied to underwriter $i$, respectively. We assume that the parameters $v_{j}$ are known, representing the profit associated with accepting the policy under review in task $j$. We inform these parameters using a separate machine-learning model, noting that insurance company may have internal models capable of estimating the expected value of accepting various policies.

Finally, we define the set $\mathcal{X}$ to ensure that the assignments accurately reflect the time-based and capacity constraints of the real-world business context. Let $T$ represent the set of days within the one-month planning horizon, and $J_{t}$ denote the policies reviewed during day $t\in T$. To maintain alignment with historical demand, we require that each task be reviewed on the same day as it was in the historical data. Additionally, each underwriter is assigned a capacity, $c_{it}$, representing the number of tasks that worker $i\in I$ reviewed during day $t$ in the historical data. We assume that each underwriter cannot exceed this daily capacity. With these constraints defined, we formulate the complete DAO formulation as follows:

maximize $$\ \sum_{i\in I}\sum_{j\in J}\sum_{m\in M(i)}\mathbb{P}(B^{m}_{ij}=1)v_{j}z_{ijm}$$ (7a)

subject to $$\sum_{m\in M(i)}z_{ijm}=x_{ij},\quad\forall i\in I,j\in J,$$ (7b)

$$z_{ijm}\leq y_{ij},\quad\forall i\in I,j\in J,m\in M(i)$$ (7c)

$$\sum_{m\in M(i)}y_{im}=1,\quad\forall i\in I,j\in J,$$ (7d)

$$\sum_{i\in I}x_{ij}=1,\quad\forall j\in J,$$ (7e)

$$\sum_{j\in J_{t}}x_{ij}\leq c_{it},\quad\forall i\in I,t\in T,$$ (7f)

$$z_{ijm}\in\{0,1\},\quad\forall i\in I,j\in J,m\in M(i),$$ (7g)

$$y_{im}\in\{0,1\},\quad\forall i\in I,m\in M(i),$$ (7h)

$$x_{ij}\in\{0,1\},\quad\forall i\in I,j\in J.$$ (7i)

This formulation follows the same structure as the DAO model presented in Section 3.2. The constraints of $\mathcal{X}$ are now explicitly defined by constraint set (7e), which ensures that all tasks are assigned to exactly one underwriter, and constraint set (7f), which enforces that the number of tasks from $J_{t}$ assigned to each worker does not exceed their daily capacity, $c_{it}$, for any $t\in T$. As noted previously, when we only consider one candidate predictive model per worker ($M(i)=1$), the formulation simplifies to the UAO model presented in Section 3.1.2.

5 Case Study Results

In this section, we present the findings of the case study. First, in Section 5.1, we examine the expected value of the solutions generated by the three UAO approaches and the proposed DAO approach when optimizing underwriter allocation. This analysis focuses on how each method for leveraging predictive insights impacts the profitability of the allocation strategy. Next, in Section 5.2, we investigate how varying the amount of training data influences optimization outcomes, highlighting the adaptability of each method across different data settings and the data-responsiveness of the DAO framework. In Section 5.3, we examine the predictive model selection of the DAO approach, exploring its implications for workforce optimization and offering insights that could inform other applications. Finally, in Section 5.4, we analyze how tasks are allocated within the DAO solution, focusing on the distribution of task values and the strategic use of different predictive models based on task characteristics.

5.1 Worker Allocation Objective Value

In our case study, we evaluated three distinct UAO approaches, each leveraging a different predictive model to forecast underwriter behavior, as well as the DAO approach, where the predictive model representing each underwriter’s behavior is dynamically selected within the allocation model. The goal was to assess how each approach, rooted in different predictive strategies, impacts the profitability of workforce allocation. The expected value of the resulting worker allocations, computed as the-percentage improvement over the expected value of the random policy, is presented in Figure 5. The expected value of a random policy is computed as the expected predicted value obtained by randomly assigning tasks to workers and predictive models, subject to the allocation constraints.

Figure 5: Expected profit increase compared to random worker allocation when different behavioral predictive models are used in the optimization framework.

The results indicate that all four optimization approaches significantly outperformed the randomized assignment policy in terms of expected profitability. Among the UAO approaches, which rely on a single predictive model for each underwriter, the aggregate model led to an 8.4-percent increase in profits, the profile model yielded a 10.7-percent increase, and the individualized model achieved a 12.1-percent increase. Interestingly, despite the aggregate model having the highest AUC (Figure 4), it produced the smallest profit improvement, while the individualized model, with the lowest AUC, delivered the highest profit increase. This highlights a key insight: while overall predictive accuracy (as indicated by AUC) is important, the ability to capture individual nuances in worker behavior is crucial for optimizing workforce allocation.

The DAO approach, which allows for the selection of the optimal predictive model for each underwriter within the optimization process, demonstrated the best overall performance, achieving a 12.9-percent improvement over the randomized policy. This prompts a key consideration as to whether the additional 0.8-percent profit gain from the DAO approach is worth the added complexity of implementing a more sophisticated model selection strategy. In many cases, the individualized models within the UAO framework captured the majority of the potential performance improvements, suggesting that this approach may be more suitable for organizations that prioritize operational simplicity over the complexities of the DAO methodology.

5.2 Impact of Varying Training Data

To further explore the adaptability of each approach, we analyzed how varying the amount of training data influences the performance of the UAO and DAO approaches. This analysis assesses how both the quantity and recency of data impacts predictive accuracy and the resulting optimization outcomes. This is an important consideration in industries like auto insurance, where conditions can change rapidly.

We evaluated the models using training data ranging from the most recent 3 to 18 months, in 3-month increments to understand the effects of data recency. Figure 6 illustrates the impact of these different training data sets on the expected value of the resulting worker allocations, computed as the-percentage improvement over the expected value of the random policy.

Figure 6: Expected profit increase compared to random worker allocation for different predictive model selections and varying amounts of training data.

The results reveal that the UAO approach using the aggregate predictive model consistently delivered an 8-percent profit increase, regardless of data size. The UAO approach with profile predictive models performed better, achieving approximately an 11-percent profit increase. With smaller datasets (3-6 months), the UAO profile model proved to be the most effective, but as more training data became available, the individualized model emerged as the top performer, reaching a 12-percent profit increase with 15 months of training data.

In all scenarios, the DAO approach outperformed the UAO strategies, with its largest improvement resulting in a 1.5-percent additional profit increase over the best-performing UAO strategy. This occurred when predictive models were trained on 9 months of data. We hypothesize that the DAO approach offers the greatest improvements over UAO at this point because, around the 9-month range, the benefit of increasing data on AUC began to plateau for all three predictive models (Appendix C). This suggests that the DAO approach is most effective when the predictive models have all reached a reasonable level of predictive power. This is likely because the models need to be sufficiently reliable to make the dynamic selection process meaningful, enabling the DAO framework to leverage the strengths of each predictive model and optimize workforce allocation effectively.

These findings align with the underlying intuition of the problem. Although the aggregate predictive model excelled in general predictive accuracy (Figure 4), it struggled to capture the specific nuances of individual worker behavior necessary for optimal assignment policies. The profile model was particularly effective with limited data, balancing general worker tendencies with sufficient data for accuracy. As the training data increased, the individualized models gained a distinct advantage, overcoming the limitations of smaller datasets. Ultimately, the DAO approach demonstrated its strength by adapting flexibly and efficiently to a range of data conditions, making it the most robust strategy across both data-scarce and data-rich environments.

5.3 DAO Predictive Model Selection

In this section, we examine the insights provided by the DAO predictive model selection. Figure 7 shows the predictive model choices within the DAO approach across varying amounts of training data.

Figure 7: Predictive model selection in the DAO approach across varying amounts of training data.

As shown, with limited data (3 to 6 months), the profile model is predominantly selected for most workers. However, as more data becomes available, the preference shifts towards individualized predictive models. This shift makes sense because individual models require more data to yield accurate results. The key takeaway from this analysis is that the aggregate model is rarely the best choice for predicting worker behavior. Even when predictive power is limited, it is better to leverage the nuances of worker behavior that are captured through more personalized predictive models. When data is scarce, the profile model effectively represents workers, but as more data accumulates, transitioning to individualized models provides a more precise understanding of each worker’s behavior, which can be better utilized in assignment policies.

5.4 Analysis of Worker Allocation Within the DAO Solution

In this section, we examine how the predictive models influence the allocation of tasks to workers. This analysis sheds light on the decision-making process within the DAO solution, particularly in how task values are distributed across workers and the role of different predictive models in these assignments. In our analysis, we examine how workers are allocated to tasks within the DAO solution, focusing on a scenario with 12 months of training data. We first analyze the value of the tasks assigned to each worker.

Figure 8 displays violin plots that show the value of tasks that are assigned to workers using either the individual, aggregate, or profile model. Note that the individual model violin plot is much larger than the aggregate model because more tasks are assigned using the individual model (see Figure 7). The median value (after standard scaling) for the individual, aggregate, and profile policies is 0.001, -1.208, and -0.328, respectively. The distribution of assigned tasks shows that higher-value tasks are typically allocated to workers represented by individualized models. Tasks assigned using profile models tend to have slightly lower values, while tasks assigned based on the aggregate model often have values closer to or even below the breakeven point. This observation highlights a clear difference in the value of tasks allocated depending on the predictive model used within the DAO framework.

Figure 8: Distribution of task values assigned to workers using individual, profile, and aggregate predictive models within the DAO framework. The red dashed line indicates policies where $v_{j}=0$.

To further understand the model assignments, we take a closer look at the acceptance probabilities of the assigned tasks. For each task, we obtain the aggregate model prediction for that task (note this value is the same for all workers). Figure 9 displays violin plots of the aggregate model prediction partitioned by tasks that were optimally assigned using either the individual, aggregate, or profile models. The median predicted probability made by the aggregate predictive model for tasks optimally assigned using the individual, aggregate, and profile models is 0.489, 0.637, and 0.576, respectively.

Figure 9: Aggregate model predictions for tasks assigned to the workers represented by individual, aggregate, or profile models.

A predicted probability near 0.5 indicates that aggregate predictive model has low confidence in it’s ability to predict worker behavior for that task. The first violin shows that when the aggregate model predictions are close to 0.5, the optimization model tends to choose individual model predictions. A similar pattern is observed for the profile violin plot. Conversely, the second violin shows that the aggregate model is most often selected in the optimization when its predicted probabilities are high, indicating aggregate model confidence and suggesting consensus across workers. This pattern highlights two key insights. First, individual and profile models are often employed when there are greater individual differences in the predictive models. This approach makes sense, as it allows the predictive model selection to effectively leverage these differences. Second, tasks with higher variance likely have higher values, which aligns with the nature of the insurance business, where policies with less consensus among workers tend to involve higher risk and may potentially lead to greater profitability (Appendix D).

These results suggest that the DAO framework strategically assigns higher-risk or more uncertain tasks to workers using individualized models, which aim to capture subtle differences in worker behavior. Conversely, tasks where worker responses are more predictable are assigned using aggregate or profile models. This indicates that employing a more personalized, individualized predictive modeling approach may be beneficial when greater variance in worker behavior is expected, even if predictive performance metrics like AUC are lower than those of less individualized models trained with more data.

6 Conclusion

This paper introduces a novel approach to workforce allocation that leverages predictive modeling and optimization to enhance worker-task assignments. By dynamically selecting the optimal predictive model for each worker within the proposed DAO framework, we improve alignment between individual worker characteristics, predictive modeling techniques, and organizational objectives. The DAO framework not only enhances decision-making accuracy but also adapts flexibly to various data environments, effectively meeting diverse organizational needs.

We present a case study based on real-world data from an auto-insurance company, offering key insights into optimizing workforce management through the application of various predictive modeling approaches within UAO and DAO frameworks. By evaluating individual, aggregate, and profile models, our analysis demonstrates that predictive insights can significantly improve workforce allocation. The DAO framework, in particular, stands out for its ability to dynamically select the most appropriate model for each worker, aligning the optimization process with specific organizational goals. This flexibility makes the DAO approach especially advantageous in scenarios with fluctuating data conditions or when the optimal predictive model is uncertain or dependent on specific worker assignments. Our findings highlight the potential of predictive modeling to refine decision-making in workforce management, driving better outcomes in complex, real-world environments.

For future research, several promising avenues exist to advance the integration of predictive models within optimization frameworks. One potential direction is to extend our proposed DAO framework by replacing the expected value objective function with stochastic programming formulations, such as Conditional Value-at-Risk (CVaR). This could provide interesting insights into how predictive model selection changes in different risk environments and with other sources of uncertainty. Additionally, we encourage future work that explores decision-aware strategies in others stages of the machine learning to optimization pipeline. By critically assessing the integration of predictive models within optimization frameworks, researchers and practitioners can develop more resilient, effective, and adaptable strategies to better address the complexities of real-world scenarios.

Support for this research was provided by American Family Insurance through a research partnership with the University of Wisconsin–Madison’s Data Science Institute.