Consistent with DHS’s acquisition policy, the Director of OTE has independently assessed major acquisition programs’ operational test results to inform acquisition decisions. Of the 49 independent assessments issued from August 2010 through December 2018, 29 were favorable and 20 were unfavorable. The proportion of favorable assessments has changed over time, in part, because programs have retested to verify corrections of previously identified deficiencies. Although the Director of OTE began assessing programs’ cyber resiliency in July 2014, few programs had conducted such testing as of December 2018, thereby limiting DHS’s insight into system vulnerabilities. OTE is taking steps to enhance programs’ capacity to conduct cyber resilience testing. DHS leadership generally used the Director of OTE’s assessments in making acquisition decisions. Programs were usually approved to progress through the acquisition life cycle, but DHS leadership frequently placed conditions on these approvals.

Assessments Provide Ratings on Effectiveness and Suitability, but Few Provide Ratings on Cyber Resiliency

Effectiveness and Suitability

From August 2010 through December 2018, the Director of OTE issued 49 letters of assessment that rated programs’ operational effectiveness and suitability.[12] We found that 29 of these assessments were favorable—meaning the Director of OTE rated program capabilities as (a) operationally effective or effective with limitations and (b) operationally suitable or suitable with limitations. The remaining 20 assessments were unfavorable—meaning the Director of OTE rated program capabilities as not operationally effective or suitable, or testing was inadequate. Figure 4 summarizes the Director of OTE’s ratings by component and appendix II provides more details on each letter of assessment.

Programs received unfavorable ratings for various reasons. In some cases, all requirements were not tested, either due to test circumstances or because they were intentionally deferred to later testing. In other cases, testing revealed that the programs did not meet requirements. Programs most frequently failed to meet the requirement for whether the system was operational when needed (known as operational availability) or it experienced critical failures more frequently than anticipated, or both.

The causes of these failures varied by program and were not always attributable to the system itself. For example, the Coast Guard’s Fast Response Cutter failed to meet its operational availability requirement in August 2013 because of issues with its main diesel engine. In contrast, deficiencies found with the Federal Emergency Management Agency’s (FEMA) Integrated Public Alert and Warning System in July 2014 were traced to infrastructure support issues with DHS and FEMA’s information technology that were external to the program.[13] Regardless of the cause, these types of failures affect when and how long a system can be reliably used by end users to complete their mission. They also require more frequent maintenance to diagnose and correct system issues, which increases program costs.

In April 2017, we reported that another reason programs failed to meet requirements during testing was because the requirements were either not written in a way that they could be tested or the desired threshold level was unachievable.[14] For example, the Director of OTE rated one of the Transportation Security Administration (TSA) passenger screening technologies as not effective in December 2016 because it failed to meet the requirement for how many passengers it could screen in an hour. The program subsequently revised the requirement to focus on the number of items screened per hour rather than passengers. In September 2018, the Director of OTE issued a memorandum confirming that the technology met the revised requirement based on a reassessment of the test data against the new definition.

Almost half (22 of 49) of the letters of assessment with ratings issued by the Director of OTE from August 2010 through December 2018 were for TSA’s passenger and baggage screening programs. These programs acquire multiple types of technologies from various vendors and each vendor’s technology must demonstrate it meets defined requirements before deployment. Of the 22 screening technologies that were operationally tested and assessed by the Director of OTE, only 10 received favorable ratings. In December 2015, we reported that immature technologies submitted by vendors was a key driver of testing failures for TSA screening equipment.[15] This was because immature technologies often experience multiple failures during testing and require multiple retests, which also leads to increased testing costs and program schedule delays since vendors need time to fix deficiencies between tests. TSA primarily planned to address this challenge by instituting a third-party testing strategy through which a third-party tester would help ensure systems are mature prior to entering TSA’s T&E process. We found that TSA had begun implementing its third-party testing strategy despite not finalizing key aspects, such as a process for approving or monitoring third-party testers or how often they would need to be recertified. We recommended TSA finalize all aspects of its strategy before implementing further third-party testing requirements for vendors to enter testing. TSA concurred with the recommendation and updated its guidance in January 2018 to ensure vendor-provided information (such as third-party test data) is sufficient to demonstrate system readiness to enter the TSA T&E process.

The proportion of favorable assessments issued by the Director of OTE has changed over time (see figure 5). For example, the proportion of favorable assessments is higher from fiscal year 2016 to 2019 (15 of 18) than from fiscal year 2012 to 2015 (10 of 25).

This change is, in part, because programs conducted follow-on operational testing to verify deferred requirements or corrections of previously identified deficiencies. Specifically, about half of the assessments (7 of 15) with favorable ratings in fiscal year 2016 or later were for programs that had previously received unfavorable ratings. For example, the Director of OTE rated the Fast Response Cutter as operationally effective and suitable in February 2017 after the program conducted additional operational testing to verify it had resolved the severe deficiency related to its main diesel engines, among other things. In addition, the Director of OTE rated a release of the Homeland Security Information Network as not suitable in December 2014 because of increased data transfer delays and unplanned outages during high system use. The Director of OTE subsequently rated the system operationally suitable in January 2016 after operational testing of the program’s full capabilities.

Retesting is not unusual, even when programs receive favorable ratings, because T&E inevitably leads to learning about system risks and areas for improvement. However, as previously discussed, the evolutionary nature of T&E encourages developmental testing to reduce risk so that programs have a higher degree of confidence that they will successfully achieve key performance requirements before initiating operational testing.

The ratings of DHS’s operational test results discussed above are comparable to those experienced by the Department of Defense (DOD) in the early 2000s. For example, the Defense Science Board initiated a study in 2007 after identifying a dramatic increase in the number of DOD systems being rated not operationally suitable from 2001 to 2006.[16] The study primarily attributed these ratings to shortfalls with system reliability and a reduction of experienced T&E personnel to perform oversight of developmental testing performed by contractors, among other things. For its part, OTE has initiated efforts to address similar issues and increase the emphasis of T&E earlier in the DHS acquisition life cycle by updating its policies and guidance and improving training for test personnel in calendar year 2017—efforts that we assess later in this report.

Cyber Resilience

While it is important that systems work when needed, cyberattacks have the potential to prevent them from doing so. Cyberattacks can target any system that is dependent on software, potentially leading to an inability for end users to complete missions or even loss of life.

The Director of OTE began assessing programs’ operational resilience to cyberattacks in July 2014, but few programs have conducted cyber resilience testing. As of December 2018, only five programs had conducted this testing to support an assessment by the Director of OTE and only two of these programs were rated as operationally cyber resilient. The remaining three programs were rated not operationally cyber resilient because system security tools did not detect intrusions and testers found significant system or network vulnerabilities, among other reasons. Appendix II provides more details on each letter of assessment.

Programs that did not conduct cyber resilience testing to support a rating by the Director of OTE have not done so primarily because it was not a consideration at the time that they initiated development, some as early as fiscal year 2002. As a result, these programs did not have operational requirements to guide the design, development, and testing of cyber resiliency into systems. In October 2015, the Director of OTE issued a memorandum that established an expectation that all operational testing include an assessment of cybersecurity since real-world events demonstrated that end users would be using systems in an environment where cyber threats would attempt to deny or disrupt their ability to carry out missions. The memorandum included procedures for planning and reporting on cyber resilience testing. For example, it directed programs to conduct a threat assessment to identify current cyber threats and testers to work with user representatives as needed to develop cybersecurity measures.

Programs’ compliance with this guidance has been slow, in part, because of the time needed to adequately plan and coordinate testing. For example, in August 2018, the Coast Guard completed a nearly 3-year effort to conduct cyber resilience testing on the National Security Cutter—the first Coast Guard asset to undergo this type of testing—as a part of follow-on operational testing.[17] The Coast Guard initially planned to conduct cyber resilience testing in fiscal year 2017, but these plans were delayed by a year because of a change in operational schedules for fielded cutters, among other things. Testing was conducted by the Navy with support from the Sandia National Lab and other DOD entities, and consisted of cooperative and adversarial assessments. The Director of OTE identified the effort as setting a standard for assessing DHS’s major acquisition programs’ cyber resilience.[18]

Although we and others have warned of cyber risks since the 1990s, determining how to build, test, and maintain cyber resilient systems is a government-wide challenge that is not unique to DHS.[19] In October 2018, we reported that DOD was just beginning to grapple with the scale of its cyber vulnerabilities because, until recently, it did not prioritize weapon systems cybersecurity.[20] We found that DOD routinely identified mission-critical cyber vulnerabilities through operational testing, but that these likely represent a fraction of the total vulnerabilities because testing did not reflect the full range of threats and not all programs have conducted cyber resilience testing.

Since few DHS major acquisition programs have conducted cyber resilience testing, the department also has limited insight into the cyber vulnerabilities of its acquired systems. Although programs are beginning to take steps to conduct cyber resilience testing, we reported in October 2018 that doing so late in the development cycle or after a system has been deployed is more costly and difficult than designing cyber resilience testing into development of the system from the beginning.[21] For example, Navy testers conducting the cyber resilience testing on the National Security Cutter after it was already operational told us there were certain aspects of the ship they could not test because it could compromise the safety of the crew or cause irreparable damage.

Some programs—including ones initiated after the Director of OTE’s 2015 memorandum—continue to defer cyber resilience testing. This approach limits DHS’s ability to understand and mitigate vulnerabilities that could be exploited by cyber threats. For example, officials from Customs and Border Protection’s (CBP) Biometric Entry-Exit Program, which was initiated in June 2017, told us that they plan to conduct cyber resilience testing after achieving ADE 3 because they needed additional time to develop a more rigorous test plan in coordination with OTE.[22] The program plans to conduct its ADE 3 review in September 2019, but had already deployed capabilities to airports by December 2018.

OTE is taking additional steps to enhance programs’ capacity to conduct cyber resilience testing. For example, in fiscal year 2018, OTE developed a training course and released supplemental guidance focused on planning and performing cyber resilience testing. OTE also awarded a contract to the Johns Hopkins University’s Applied Physics Laboratory for support, which included performing cyber resiliency testing on DHS major acquisition information technology programs. The results of this testing are intended to further OTE’s ability to develop comprehensive, department-wide policy and procedures. Since DHS is still implementing cyber resilience testing and developing appropriate policies and procedures, we will continue to track the department’s progress through our ongoing assessments of major acquisition programs.[23]

DHS Leadership Generally Used Assessments to Make Informed Acquisition Decisions

Consistent with DHS’s acquisition policy, the Director of OTE’s letters of assessment informed DHS leadership’s acquisition decisions. We reviewed the 38 acquisition decision memorandums issued after the 49 operational test events for which the Director of OTE provided a letter of assessment with ratings.[24] Programs generally received approval to progress through the acquisition life cycle based on the Director of OTE’s ratings as one of the factors considered in decision-making. In more than half of the cases we reviewed (26 of 38), DHS leadership placed conditions on its approval or directed programs to address issues discovered during testing. For example:

• In March 2011, DHS leadership granted ADE 2C approval with conditions for an explosives detection system acquired by TSA’s Electronic Baggage Screening Program. This system received an unfavorable assessment from the Director of OTE primarily because it had not been used enough during operational testing to support a rating of operational effectiveness or suitability. DHS leadership limited TSA’s procurement to 28 systems and required the program to conduct additional operational testing and receive another letter of assessment from the Director of OTE before it could procure more systems. The program completed follow-on operational testing of the system and, in September 2011, the Director of OTE rated it operationally effective and suitable with limitations.
• In January 2015, DHS leadership granted CBP’s Multi-role Enforcement Aircraft program conditional approval to acquire two aircraft. The Director of OTE could not assess operational suitability because testing was limited and did not include testing of availability, reliability, and maintainability, among other things. Leadership’s approval was contingent on the program completing a number of actions, including preparing a plan to correct issues identified during the operational test and providing a status of corrections to the Director of OTE. The program conducted additional operational testing in July 2015, which received favorable results. Although test data showed the aircraft’s availability was lower than required, the Director of OTE rated the aircraft as suitable with limitations because the availability rating had improved after testing concluded.

In our review of acquisition decision memorandums, we found a case where a program received an unfavorable assessment from the Director of OTE and DHS leadership did not let the program proceed as planned. The FEMA Logistics Supply Chain Management System conducted initial operational testing in calendar year 2013, which the Director of OTE determined was inadequate to support ratings for operational effectiveness and suitability.[25] In response to this and other factors, DHS leadership paused the program in April 2014 and directed officials to re-evaluate its development strategy. For example, the program was directed to (1) revisit its requirements and identify capability gaps based on the operational test results and the Director of OTE’s letter of assessment; and (2) conduct an analysis of alternatives for addressing identified capability gaps.

In March 2016, DHS leadership approved this program to resume development after completing these actions, but also required the program to take additional steps to address recommendations made by the Director of OTE. For example, the program was required to select a new operational test agent and update its T&E master plan to address issues identified in the letter of assessment before conducting additional operational testing. The program completed follow-on operational testing in June 2018, which the Director of OTE rated as operationally effective and suitable with limitations because the system did not have an off-site backup server to quickly restore operations in the event of a catastrophic failure of the primary server. Based on these results, DHS leadership did not approve the program’s ADE 3 or acknowledgement of full operational capability in February 2019, and directed the program to implement a backup server solution by the end of August 2019.

We assessed the T&E policies and guidance DHS issued between May 2017 and January 2019 and found that they generally reflect key practices. For almost two decades, we have reported that successful acquisitions engage in a continuous cycle of improvement by conducting T&E throughout development and incorporating lessons learned. For example, in July 2000, we examined the practices that private sector entities used to validate that a product works as intended and determined that leading commercial firms viewed T&E as a constructive tool throughout product development.[26] Specifically, we found that leading firms ensure that (1) the right validation events—tests, simulations, and other means for demonstrating product maturity—occur at the right times; (2) each validation event produces quality results; and (3) the knowledge gained from an event is used to improve the product. By holding challenging tests early, firms exposed weaknesses in a product’s design and limited design changes late in the development process when it is harder and more expensive to address issues.

Based on this and our other work, as well as a review of related third-party and other government entities’ reports, we developed a list of key T&E practices that consist of 17 elements that contribute to successful acquisition outcomes.[27] The list of practices we developed is not exhaustive and is intended to be at a level high enough so that they may be applied to assess various types of acquisitions—including both hardware and software—regardless of the developer or development approach.

We used these key practices to assess DHS’s T&E policies and guidance since they similarly provide a high-level framework for outlining what is expected of major programs throughout the acquisition life cycle. In May 2009, DHS issued its first directive requiring major acquisition programs to ensure adequate and timely T&E is performed to support informed acquisition decision-making and outlining oversight of those activities. OTE has since updated the directive and developed additional guidance to increase the emphasis of T&E earlier in the acquisition life cycle and to address issues programs have encountered during operational testing, among other things. For example, OTE updated the directive in May 2017 and developed an accompanying instruction in July 2017 that clarified the roles and responsibilities for certain T&E activities within the department and at certain acquisition milestones. OTE also released supplemental guidance on more detailed topics, such as incorporating T&E into programs’ contracts and evaluating threats to ensure testing reflects realistic scenarios.

Table 1 summarizes our assessment of OTE’s policies and guidance against our key T&E practice areas and elements.

As reflected in the table, OTE’s policies and guidance met the elements of our key T&E practice areas in nearly all cases. For example:

• Develop a test strategy to demonstrate program requirements. OTE’s policies require programs to document a test strategy for verifying program requirements in a T&E master plan, which supplemental guidance indicates should outline opportunities for integrated testing, incorporate measures of success, and identify any potential limitations, among other things. Integrated testing is intended to increase efficiencies by collecting data through one test event that supports the evaluation needs of multiple stakeholders, such as developmental testers and operational testers. OTE also enhanced its guidance on how programs should plan for early reliability testing because it considers this metric to have the greatest effect on system performance, which includes how frequently the system fails. This may help programs address reliability problems before initiating operational testing which, as we previously mentioned, is a reason DHS’s major acquisition programs frequently received unfavorable test ratings from the Director of OTE. OTE’s guidance also states that a program’s T&E master plan should be based on credible threat information to ensure testing is conducted under realistic scenarios. In July 2018, OTE released additional guidance focused on cyber threats that provides information on how programs can identify potential threats, evaluate their impact, and document these threats.
• Identify and secure resources to conduct testing. OTE’s July 2017 instruction directs program managers of major acquisitions to designate a T&E manager who is required to be level III certified in the T&E career field. The T&E manager is to coordinate the planning, management, and oversight of all T&E activities; lead the development of a program’s T&E master plan; and coordinate test resources, among other duties. We assess DHS’s implementation of the certified T&E manager requirement later in this report.
• Conduct testing throughout the acquisition life cycle. OTE’s guidance states that programs generally should conduct a review to assess component or subsystem test results prior to beginning system integration and comprehensive developmental testing. In addition, prior to full deployment (e.g., ADE 3), OTE’s instruction directs programs to conduct operational testing of the complete system in a realistic environment.
• Use test results to inform decisions. OTE’s updated directive added a phase of independent testing for any program that has a limited production decision (e.g., ADE 2C). This decision is optional depending on the program’s development approach. But, if conducted, the Director of OTE now issues a letter of assessment prior to ADE 2C to inform the acquisition decision authority about the program’s performance. Previously, the directive only called for the Director of OTE to issue a letter of assessment prior to ADE 3 decisions.

We found that OTE’s guidance only partially met the key practice element to test that components and subsystems work together as a system in a controlled setting before finalizing a system’s design. OTE’s guidance instructs programs to conduct integration testing on components and subsystems. However, it calls for this type of testing to occur after the critical design review, which is when the design is finalized. This increases the risk that programs will need to make changes to system components after the critical design review, which our past work has shown can cause cost increases or schedule delays.[28] OTE officials acknowledged that the current T&E guidance could be clearer on when integration testing should occur in the development process. These officials told us that they intend to adjust their policies and guidance, which they are updating to align with the May 2019 acquisition management instruction. Until OTE updates this guidance, programs could experience costly design changes resulting from conducting integration testing late in the acquisition life cycle.

We found that DHS’s T&E training reflects most attributes of an effective federal training program, but does not fully reflect attributes pertaining to assessing the benefits of training. In March 2004, we issued a guide for assessing federal training programs that breaks the training and development process into four broad, interrelated components—(1) planning and front-end analysis, (2) design and development, (3) implementation, and (4) evaluation—and identifies attributes of effective training and development programs that should be present in each of the components.[29] These components are not mutually exclusive and encompass attributes that may be related. For example, each component includes attributes related to assessing training and development benefits rather than these attributes being confined solely to the evaluation component.

OTE officials told us they assumed responsibility for providing instructors and for updating the T&E training materials from the Office of Chief Procurement Officer (OCPO) and Homeland Security Acquisition Institute (HSAI) in 2017. OTE worked with OCPO and HSAI to implement changes intended to ensure the training met workforce needs and reflected current policies and guidance. For example, OTE updated the content for each of the certification level core T&E courses—which were originally based on DOD’s T&E training—to reflect DHS’s acquisition process, among other things.

As summarized in figure 6, OTE’s T&E training, particularly the core T&E courses, either met or partially met the relevant attributes from our guide. A more detailed description of our assessment methodology is presented in appendix I.

OTE’s T&E training reflects attributes of effective training in each of the four components of the training and development process, such as:

• OTE coordinated with OCPO and HSAI to determine the skills and competencies needed for an effective T&E workforce, which served as the foundation for developing the learning objectives and content for the core T&E courses.
• OTE partners with HSAI to provide formal instruction for in-person courses at HSAI’s training facility or online courses through the Defense Acquisition University. OTE supplements these courses with informal training opportunities, such as seminars and workshops that are tailored to meet component and program needs or focus on specific subject areas. For example, OTE has sponsored seminars dedicated to writing an effective test strategy and facilitating a table-top exercise to inform a program’s cyber resilience testing. OTE also hosts an annual symposium for T&E managers to facilitate knowledge sharing across the department.
• OTE solicits and incorporates stakeholder feedback from the T&E Council and T&E Working Integrated Product Team into the planning, design, and implementation of its training efforts to ensure that the training addresses DHS’s workforce needs. Primarily, OTE uses these groups to review existing course content, determine the frequency of course offerings, and identify topics for new courses. For example, OTE developed a course specifically for operational test agent leads after the T&E Working Integrated Product Team identified a gap in training for those who plan and conduct operational testing for DHS’s major acquisition programs. As of August 2018, the Director of OTE requires the operational test agent lead for major acquisition programs to complete this course prior to commencing operational testing.
• OTE uses a service contract to fulfill its need for classroom instructors for the core T&E courses. According to OTE officials, this allows them to provide full-time instructors with T&E experience, which provides consistency across the trainings. OTE’s Deputy Directors and Test Area Managers also present on certain topics during the core T&E courses to provide knowledgeable expertise. In addition, OTE has secured other experts to serve as guest speakers at various trainings, seminars, and workshops, as well as to assist programs with specific needs. For example, OTE partnered with the Air Force Institute of Technology to establish the DHS Scientific Test and Analysis Techniques Center of Excellence to sponsor workshops and a module in the intermediate core T&E course dedicated to increasing the use of statistics and other tools to support T&E, and to provide select acquisition programs assistance with developing an analytical approach and test design.

The six attributes that OTE’s T&E training partially reflects are all related to assessing the benefits of the training. OTE uses stakeholder feedback from the T&E Working Integrated Product Team and student feedback from course evaluations to improve training content and delivery. However, OTE has not yet taken steps to assess whether the training offered has led to improved organizational results. For example, OTE has identified better test documentation and more favorable operational test results as desired outcomes. But it has not established performance measures to determine how the training may contribute to achieving these outcomes. According to officials, OTE has not yet assessed its training because it prioritized developing and delivering courses since assuming responsibility in 2017. Additionally, OTE has spent approximately 30 percent of its annual budget on training since 2017 and reviews its proposed training expenses annually, but has not evaluated its return on this investment. OTE officials acknowledged that establishing performance measures would be beneficial, but said it would be challenging to reliably assess against those measures. Nevertheless, by developing an assessment process that includes performance measures for T&E training, OTE can better ensure that it is offering effective training that addresses training objectives and achieves desired results.

We found that DHS faces challenges with its T&E workforce to effectively provide oversight at the program and headquarters levels. Specifically, most programs do not have a level III certified T&E manager, despite the requirement for program managers to designate one for major programs. Additionally, OTE’s data for monitoring whether programs have a certified T&E manager are unreliable. Finally, DHS has expanded OTE’s responsibilities for T&E oversight in recent years, but has not assessed whether OTE has the workforce needed to fully execute these responsibilities. OTE officials said they have had to prioritize their oversight efforts, making it difficult for the office to fully execute its responsibilities across the range of DHS major acquisition programs.

Most Programs Lack a Certified Test and Evaluation Manager and DHS Policy Unclear

As previously mentioned, OTE’s July 2017 instruction directs program managers of major acquisitions to designate a T&E manager who is required to be level III certified in the T&E career field. As of March 2019, only 17 of 43 major acquisition programs we reviewed reported to us that they have a level III certified T&E manager (see figure 7).

Officials from select programs that have designated a T&E manager who does not yet meet the certification requirement indicated they are working toward achieving level III certification. For example, officials from the CBP Non-Intrusive Inspection Systems program reported that the designated T&E manager was unable to complete his final course to achieve level III certification because it was cancelled as a result of the partial government shutdown in January 2019.

Officials from the 12 programs without a designated T&E manager provided various reasons for not having one, including:

• The position is vacant because the program office reorganized or the designated T&E manager left the agency. For example, officials from FEMA’s Grants Management Modernization program reported that the designated T&E manager moved to another agency so the role is being temporarily filled by service contractor personnel with prior T&E experience until they can hire a qualified, permanent replacement.
• The program manager determined the program did not need a T&E manager based on its development approach. For example, officials from CBP’s Automated Commercial Environment program reported that the program manager did not designate a T&E manager because of the program’s shift to agile software development.[30]
• The program manager determined the program was exempt from the certified T&E manager requirement based on where the program was in the acquisition life cycle. For example, officials from Immigration and Customs Enforcement’s U.S. Code, Title 8, Aliens and Nationality program said they were too early in the acquisition life cycle to designate a T&E manager. On the other hand, officials from the Coast Guard’s Nationwide Automatic Identification System program said they did not need one because they were later in the acquisition cycle, past ADE 3.

As of April 2019, OTE officials confirmed they had not exempted any program from designating a certified T&E manager.[31] OTE officials acknowledged that some programs that are past ADE 3 may not need a T&E manager depending on the program’s acquisition strategy and whether it had successfully completed testing. However, they cautioned that programs may continue to need T&E support late in the acquisition life cycle since many add capabilities after deployment and assessments against emerging cyber threats should be ongoing.

OTE officials told us they expect programs to designate a T&E manager early in the acquisition life cycle—no later than ADE 2A—to assist with developing system requirements that are measurable, testable, and achievable and establish a sound test strategy. This expectation is consistent with DHS’s May 2019 update to its acquisition management instruction, which requires programs to submit an initial T&E master plan earlier in the life cycle—at ADE 2A compared to ADE 2B under the prior instruction. OTE officials emphasized this early designation is critical for programs using agile software development because testing is conducted early and often by the various development teams. OTE officials said they communicate their expectation through direct interaction with programs and during Acquisition Review Board meetings and other forums.

However, OTE’s policy only indicates program managers should designate a T&E manager “as early as practicable.” Specifying in policy when programs should designate a T&E manager would make OTE’s expectation clear that program managers are to identify knowledgeable T&E personnel no later than ADE 2A, which is where important decisions about the program’s requirements and test strategy are now made.

Data for Monitoring Program Test and Evaluation Managers Are Unreliable

OTE internally tracks data on programs’ T&E managers, but we found the data to be unreliable. In comparing the information we compiled on T&E managers (as discussed above) against OTE’s internal data, we found that OTE had inaccurate information for about half (19 of 40) of the programs.[32] In most cases, OTE either identified a T&E manager when a program reported not having one, identified the wrong individual in the position, or identified the wrong certification level. For example, OTE identified that CBP’s Integrated Fixed Tower program had a T&E manager as of December 2018, but the program reported that the position was vacated in October 2018 and remained unfilled as of February 2019. Moreover, the individual identified by OTE as the T&E manager did not match the former T&E manager identified by the program.

OTE officials said the Deputy Directors and Test Area Managers are responsible for inputting and maintaining the data for their assigned programs in the OTE internal tracker at least quarterly or as changes occur (e.g., following an ADE or test plan approval). Service contractor personnel for OTE then perform a quarterly review for missing information and to verify the T&E manager certification levels against a list provided by HSAI.

However, OTE has not established an internal control process for ensuring the accuracy of all the data in its internal tracker. Standards for Internal Control in the Federal Government advises management to obtain relevant data—and evaluate the reliability of the data—in a timely manner so that the data can be used for effective monitoring.[33] By establishing an internal control process for collecting and maintaining its data, OTE can improve its ability to accurately track programs’ T&E managers.

Full Scope of Test and Evaluation Oversight Responsibilities Not Being Executed with Current Workforce

DHS has expanded OTE’s responsibilities for T&E oversight in recent years. Specifically:

• The Deputy Secretary of DHS updated the department’s delegation of authorities to the Director of OTE in June 2016. This delegation extended the Director’s oversight to include developmental testing as reflected in major acquisition programs’ T&E master plans.
• OTE officials told us in February 2019 that the senior official performing the duties of the Under Secretary for Science and Technology requested that OTE begin providing oversight of T&E activities conducted by the directorate’s research and development projects. This effort is to include developing T&E policies and procedures for research and development, as well as providing formal support to specific projects that is similar to that provided to major acquisition programs.
• As previously noted, the May 2019 update to DHS’s acquisition management instruction requires programs to submit an initial T&E master plan earlier in the life cycle. This will require OTE staff to begin engaging with programs earlier to support development and approval of this plan. ecifically, ated duties with respect to

OTE officials stated that executing the office’s expanded oversight responsibilities is difficult because the Science and Technology Directorate has not authorized changes to OTE’s federal workforce since 2014. OTE has awarded contracts for support in developing policy and guidance, facilitating trainings, and providing technical expertise in key areas—such as reliability and cybersecurity—among other things. While this expanded the capacity of the office, it also added responsibilities on OTE’s staff since it must oversee work completed by contractors.

Despite these efforts, OTE officials told us they have to prioritize support to those major acquisition programs that are of high importance to the department and actively engaged in planning or conducting operational testing. As a result, OTE is unable to fully meet the additional delegated duties for developmental testing oversight. Specifically, in April 2019, officials stated that they did not have the capacity to help shape developmental test plans, observe testing, or provide feedback on test results in the same manner that they do for operational tests. While OTE officials plan to award a service contract for conducting activities in support of research and development T&E, the management and oversight of these contracted activities will increase the administrative duties on OTE’s existing federal staff.

Workforce planning helps an organization align its human capital, both federal and contracted, with its current and emerging mission and programmatic goals. In December 2003, we identified several key principles for effective strategic workforce planning.[34] These principles include:

• Determining the critical skills and number of employees needed to achieve programmatic results;
• Identifying and developing strategies to address staffing and skills gaps; and
• Monitoring and evaluating progress toward human capital goals.

DHS did not assess OTE’s human capital before expanding the office’s oversight responsibilities. By assessing OTE’s workforce, the Science and Technology Directorate can take an important step to ensure that OTE has the appropriate number of staff with the necessary skills to fulfill the full scope of its expanded oversight responsibilities.