With so much software being developed by outsourcing providers and contractors that are sometimes halfway across the world, organizations must have a well-thought-out strategy to ensure that the software that is developed for them is free from security vulnerabilities and in line with their desired level of security assurance. As with most initiatives, this set of activities must have as low an impact on costs and schedules as possible.
In my last article, I discussed what it takes to build a software security program.¹ The guidance in that article will serve you well throughout the product lifecycle as your organization designs, develops, and deploys software. However, many organizations these days develop their software by making heavy use of outsourcing and contractors. The cost benefits of doing this have made it an especially popular option given the recent economic climate.
However, outsourcing can have a significant impact on the security of applications developed by your organization. Hence, one of the most common questions we at Foundstone field when working with large organizations goes something like this: "Seventy percent of our software development at this point is outsourced. How do we ensure that our outsourced partner/contractor performs the appropriate levels of due diligence for security?" With the volume of software being developed by contractors, the answer to this question, as one would imagine, can very often be the difference between secure software and software that is inundated with vulnerabilities, backdoors, and other security policy violations. Indeed, the rising popularity of the cloud computing paradigm in general and Software as a Service (SaaS) in particular has only added to the trend. These models not only outsource the development of business-critical applications but also host these applications - and perhaps most importantly, the data within the applications, which is often of great value to your organization.
It is therefore vital to establish a process for ensuring that the applications within your organization are securely developed irrespective of who is developing them - whether a full-time employee, a contractor, or an outsourcing vendor. Based on our experience, we offer solutions that, while not entirely painless, can significantly improve the security quality of your applications and yet not result in large cost or timeline overruns or a troubled relationship with your vendors.
Vendor Management
A number of compliance regulations and best practice standards, such as the PCI-DSS² and ISO, require the creation of a vendor management program. Such a program is meant to ensure that appropriate levels of attention are paid to third parties that can heavily influence the security of your systems. We recommend extending that approach to software development relationships as well.
Essentially, if a vendor is involved in development, deployment, or hosting of your applications, that vendor should be covered within such a program. The most common examples of such vendors are your outsourced providers, but it is also important to consider developers of third-party components, such as user interface grids or API developers that provide access to data on mainframes. Similarly, it is important to consider contractors who work within your own environment just like your full-time employees, as well as hosting and SaaS providers.
Once you have a list of who must be covered by a Software Security Vendor Management Program, the next task is to apply a standard process to them to ensure that your organization can achieve the level of security assurance it needs, even when engaging contractors. It helps if you think of this process as part of a three-pronged framework. The focus of this framework is to associate key software security aspects with the nature and stage of the relationship that exists between the contractor and your organization. The three key areas that must be considered are:

• Contracts
• Validation
• Operations

Let us examine contracts in detail. (We'll cover validation and operations in a future article.) Before we dive too deep, however, it is important to state, as always, that organizations are best served by taking a risk-based approach to this framework and enforcing it. This means that while this article describes a large number of activities and proposals, not all of them may be appropriate to your organization or to the specific project you are currently working on.
In some cases, for example, if the vendor is developing software that will store patient information or personally identifiable information (PII), the framework might be significantly more stringent than in cases in which the application being developed has little or no security significance. In fact, the ability to adapt has to be a key design criterion of any such framework, since a one-size-fits-all solution is likely to be no solution at all.
Before we delve into specific details, let's examine one other order of business: the disclaimer. The information provided within this article represents guidance and experience from the author. It should not be construed as legal advice, and I strongly recommend that your organization consult either the internal legal group or qualified outside counsel before using any of the thoughts and ideas that follow.
Contracts
Contracts are often the first place in which an organization can establish standards for software security with its vendors. That's because it's also where the organization has the most leverage with contractors as they strive to win its business. Unfortunately, more often than not, we find that organizations are lax in defining any requirements or metrics around software security and thus miss out on the opportunity to influence the relationship. This often happens because the individuals tasked with software security within an organization are rarely involved in the contracting process; the procurement and legal personnel involved in this process rarely have the background necessary to mandate security.
As a solution to this issue, it is therefore necessary that very early in the contracting workflow - e.g., as part of the RFI (Request for Information) or RFP (Request for Proposal) process - security requirements at a process level must be defined and included within the larger requirements specification shared with the vendor. Given the increase in regulatory oversight, this is no longer just a nice-to-have; it's required as part of your organization's compliance objectives.
The most common approach organizations have taken with including security language in contracts is to include an annex or appendix that lists concrete requirements and standards that need to be met.
Training Personnel
One of the key aspects to cover within your contracts is the quality of personnel assigned to your project. Approaches to this could range from detailed background checks on all vendor staff involved in the project to assessments of individual skills. It is common to require certain basic qualifications in the form of certifications, formal educational degrees or diplomas, or often - in the case of software security - completion of a specific training curriculum.
Some companies have taken a different approach and have trained selected vendor staff themselves. While this is an excellent strategy, it does have one major drawback that a number of organizations complain about: they often find that once they have invested in training contractors, those contractors end up moving on to other projects and other customers as their value increases and other opportunities come their way.
It can be extremely frustrating for an organization to spend both time and money teaching an individual the fundamental principles of software security - only to find that when the project eventually begins, the individual in question has been replaced by someone who is untrained in such matters. This means that the organization will have to go through the same cycle all over again.
Obviously, this is not a very scalable model, and therefore organizations are often better served by mandating such training as a requirement for vendors and their staff before they are even assigned to a project. In this way, the responsibility for providing the training falls on the shoulders of the vendor and, hence, risk is transferred.
Another approach is to develop customized computer-based training that focuses on specific areas within software security that are relevant to your organization. Some organizations have even gone to the extent of testing assigned staff as part of this process. In a sense, this is not much different from the onboard training that is often provided to new employees. The idea is to make them aware of your policies and expectations.
Auditing
Another issue that must be covered during the contracting process is your organization's right to audit - a right that vendors may be hesitant to accept, but again, one that is best tackled during the contracting process. Auditing is extremely difficult to do without the cooperation of the vendor.
In some cases, vendors may worry that intellectual property may be leaked to your organization. This is especially true when the vendor is selling shrink-wrapped software or SaaS. In these cases, typically, your organization only gets usage rights to the application and cannot view the source code, or oftentimes even test the application, without violating copyright laws. In such situations, especially, the vendor must agree and cooperate with testing procedures. Of course, this implies that we must define testing itself.
Depending on the perceived risk of the application and the posture of the organization, testing might range from a simple questionnaire-based risk assessment to a physical security assessment, especially when the application is being hosted by a third party. In addition, from an application security perspective, organizations might mandate some combination of threat models, code reviews, and application penetration tests.
Finally, there is also a need to document the frequency of testing, especially when this is intended to be a long-term relationship that spans multiple years and multiple versions of the application in question.
One issue that your organization might run into when negotiating the right to audit is that the vendor might have already completed an audit on its own. It is important, therefore, to mandate that such an audit must be completed by a reputed firm³ against a well-defined and accepted methodology, such as the OWASP Testing Guide⁴ or the OWASP Application Security Verification Standard.⁵ It is also possible that the vendor has been assessed as part of its own compliance needs with standards such as PCI-DSS. In these cases, it is common for organizations to require that an executive summary of the testing be provided to them.
Of course, performing testing is only one piece of the puzzle. It is also vital that the vendor actually fix the issues uncovered. It is therefore important to mandate the types of issues that must be fixed before going to production and those whose risk is acceptable. Again, your compliance objectives can help guide this decision. For instance, the PCI-DSS uses a five-point scale, in which issues marked as levels 3 through 5 must be corrected to stay in compliance. Your organization could employ a similar approach - quantitative risk models, such as CVSS,⁶ can be extremely helpful, since they attempt to remove subjectivity from the issue.
Organizations with a more stringent policy, or those dealing with particularly sensitive applications, might take the approach that every identified vulnerability needs to be fixed, and that once that is done, a retest may need to be performed and a clean report provided. Others may only require that a plan be in place for medium- or low-risk issues. Again, these are decisions that your organization will need to make based on the specifics of your risk profile.
One contentious issue when it comes to fixes to security defects is that a number of outsourcing vendors treat them as change requests and hence will charge your organization for fixing them. To use a real-world analogy, this is the equivalent of selling you a car with faulty seat belts and then charging you to replace or repair them. It is vital, therefore, that this issue be tackled within the contracting process as well. In this way, surprises can be avoided once delivery begins.
Another aspect of testing goes beyond the application itself to focus on the processes the vendor follows as well as the underlying infrastructure. Organizations might need to perform due diligence, for instance, to ensure that the data center within which the application will be hosted has appropriate levels of physical security as well as disaster controls.
Additionally, the client firm might need to require an assessment of the security of infrastructure components, such as source code control systems or management networks. This is especially true when the vendor works with a wide variety of customers, some of which might be your competitors. The vendor's policies for performing security activities, such as patching or incident response, might also be of interest based on the criticality of the application.
In short, as you define assessment requirements within the contract, it is important to consider all the different forms of testing that are necessary from your organization's security assurance perspective.
Setting Expectations
The contract is also an opportunity for your organization to set very clear expectations around the security bar that the vendor will be expected to meet. This is best done by including specific policies and procedures that the vendor must follow at all times. For instance, if the application being developed is going to handle PII from your customers and these customers share that information with you based on your privacy policy, it is critical that this policy be shared with the vendor and that the vendor read and acknowledge the same. Similarly, if your organization has a data classification policy that governs how different types of data are to be handled in storage and transit, then it is imperative that this policy be provided to the vendor if any of these data elements will be shared with them during the course of the project.
From a software security perspective, it is important to document specific coding standards that developers at the vendor are expected to follow. These must be prescriptive and provide detailed technical requirements. For instance, it is not sufficient to say strong encryption must be used. Rather, a standard must specify the algorithm, key length, and other such parameters. This leaves less to opinion and subjective analysis and is also easier to audit.
In some organizations that have mature software security programs, it is not uncommon to see these standards expand to include actual code snippets, or even to include libraries and APIs that the vendor is required to use for specific purposes, such as cryptography and data validation.
One final issue to cover within the contract is warranties. These are intended not to provide the money-back guarantee we might be used to in the real world, but rather to transfer liability. For instance, vendors are often required to warrant that the application developed or provided to your organization is free from intentional malicious code or backdoors; penalties or consequences should be spelled out if this is later found not to be the case.
Additionally, vendors are often required to provide detailed documentation. From a security perspective, we would want this documentation to include (as appropriate) artifacts such as hardening guides as well as application-specific security information. Examples of this include risk-mitigation steps that might be taken as well as information on the authorization model and how it is enforced.
To conclude, we would be remiss without referring to some of the best work done in this area that is available in the public domain - the OWASP Legal Project.⁷ This project includes sample contract language as well as other reference information on the topic that will serve you well as you attempt to implement some of the advice provided here.
Rudolph Araujo serves as a principal software security consultant and trainer at Foundstone Professional Services, a division of McAfee.
¹ www.softwaremag.com/L.cfm?Doc=1224-9/2009
² www.pcisecuritystandards.org/index.shtml
³ Understandably the term "reputed firm" is subjective. Unlike many other industries, there aren't always analyst reports that rank security consulting organizations. Hence, our recommendation is to go by the organization's reputation, presence across the country or globe, publications (books, articles, and whitepapers), and participation in major conferences. Remember that the key objective is to ensure that the firm employed by your vendor is likely to do a thorough job of testing the security of its application(s).
⁴ www.owasp.org/index.php/Category:OWASP_Testing_Project#OWASP_Testing_Guide_v3
⁵ www.owasp.org/index.php/ASVS
⁶ www.first.org/cvss/
⁷ www.owasp.org/index.php/OWASP_Legal_Project

One often-overlooked but key activity in project management is the work breakdown structure (WBS). To be able to deliver the goals or objectives of a project, it is imperative to have knowledge of what it takes to meet those goals. That's what the WBS does. Insufficient time spent performing this activity - or omission of it altogether - can doom a project.
Any project whose end game includes "last-minute activities" is a project in which the WBS has likely been ignored or not completed. These forgotten items will probably result in cost overruns as well as late deliveries and poor quality, because without a well-constructed WBS, tracking the project's duration, costs, and actual progress becomes a guessing game.
The WBS is sometimes confused with a Bill of Materials (BOM). The BOM describes the raw materials, subassemblies, components, and so on that are required to complete a project. Like a WBS, a BOM is hierarchical, with upper levels (systems) decomposing into subsystems and components. But that is where the similarities stop. A WBS is a hierarchical list of the activities involved as well as the deliverables that need to be produced in order to deliver a project to successful fruition.
A WBS can further be defined as the following:

• A product-oriented hierarchy composed of hardware, software, services, data, and facilities.
   
• A description of the product, or products, to be developed and/or produced; in so doing, it relates the elements of work to one another and to the end product.
   
• An expression of an "organic" hierarchy down to any level of concern.

The Department of Defense recommends three levels for this hierarchical decomposition; we recommend significantly more detail, because the result of our effort allows day-to-day management of our various products and development deliverables. Additionally, breaking down to a detail level facilitates the identification of risks associated with the project deliverables and task dependencies. (See Table 1.)
As a template, MIL-HDBK-881 is not exact for our purposes (we might, for example, add simulation activities that our organization or customer may deem necessary to ensure product quality), but it is a good place to begin. It gives us a running start and provides a checklist that will make it easier to remember all of the essential cost centers. Recognizing cost centers facilitates the identification and tracking of work not directly within the project manager's control - such as outsourced work packages.
Direct Reflection of Requirements
The WBS approach installs the voice of the customer as the driving mechanism for the scope of the work, which, in turn, drives the requirements. It does not matter if the requirements come from external sources or if they are internally derived. When requirements change, the WBS must also change, because of its nature as a functional decomposition of top-level deliverable elements. Once we have a structure that is visible, updating, re-planning, re-estimating costs, and task duration are simplified, because all of the elements are visible as well.
The WBS is important to project management styles and line management because the cost centers are derived from deliverable elements of the project (so much so that it is easy to develop templates for the WBS). The WBS concept works because products are composed of systems, which in turn are composed of subsystems and then components and so on. If we start with a top-level assembly as the first or second level, we can easily break the product down into "atomic"-level tasks.
The same approach will apply if we are dealing with other deliverables, such as internal specifications, models, failure mode and effects analyses, and the whole round of documents required by any formal quality system.
Tracking updates to the project scope or changing deliverables to meet requirements is where many projects go astray. Lack of change management or incompetent configuration control makes it difficult to compare what we have with what we expected or needed. With scrum, we derive the product backlog directly from the WBS. And the WBS is not simply an action item list; it's a formal document designed to support cost and schedule reporting as part of a contract. From the WBS, we can derive our action item lists, schedule, and budgets.
How Deep the WBS Should Go
The WBS will deconstruct as far as we need to go, so that we can put items into our planning documents - formal project management tool, spreadsheets, or action item lists. If we have an especially short planning horizon, as is used in scrum project management, then we have to have tasks that can be accomplished during that period. We call this highly detailed analysis "atomic" decomposition, because we are decomposing the higher-level tasks until further decomposition no longer adds value. When we complete this task, we will have a list of "atoms" that become part of our other planning documents.
In some cases, the "atoms" will be small enough that we can complete them within minutes. In our experience with a production testing group, this approach removed the excuse that "we didn't have time to do this work." By putting these small tasks into a management-supported list, we can drive some level of accomplishment through their completion.
Task Decomposition to "Atomic" Level
As we have indicated, "atomic" decomposition occurs when we take a WBS down to the level where task completion becomes binary - either it is done or it isn't done, eliminating the estimation of "percent complete." If we are really astute, we may even break our tasks down to the point where they have roughly equivalent durations, which allows for easy planning. The reasons for doing this are manifold:

• We provide immediate gratification, and thus reinforcement to our project team.
   
• We can roll up completed tasks into a standard project management program and get a "percentage complete" result, suitable for passing on to management.
   
• For agile project management methods, we can easily insert these low-level tasks into even the shortest sprint backlog.

We use this approach to WBS decomposition to ensure completion of all tasks. The cost center decomposition ensures that we identify dependencies and include them in our planning documents. Our goal is to remain flexible and lightweight, rather than burdening ourselves with a lumbering, uninspired approach to our task list.
Activity Sequencing
Each of the building blocks may have interactions and dependencies; for example, we must design hardware before we build hardware. The WBS provides enough information to define all required tasks - the project manager will still use a network diagram to represent task dependencies. The WBS does not put the dependencies together for you as part of the work.
However, breaking down the tasks to a sufficient level makes it possible for you to work through the dependencies more efficiently - since all of the tasks are identified in the WBS. This manipulation of the tasks happens after the WBS is produced and the activities are sequenced.
This approach has the advantage of eliminating the use of percentages to represent completion status, a technique that our experience shows to be overly optimistic. The small-element approach also makes for easier deployment of estimating techniques, such as Program Evaluation and Review Technique (PERT) and Monte Carlo simulation.
Time-Phased Budget
The time-phased budget will include the schedule and cost (usually in hours). This approach is necessary for Earned Value Management (EVM), which includes the following metrics:
Schedule performance index Cost performance index Cost variance Schedule variance Estimated budget to complete Estimates at completion
The WBS elements make up the accumulated project budget. (See Fig. 1.) When the time is beyond the WBS element, then the project is in duration overrun. The smaller the task, the quicker it is to identify when the project is in this overrun condition.
When a project has been deconstructed to a sufficient level of detail, and allows for early detection of schedule and budget slipping, it results in the cumulative project expenditure via WBS with a very high resolution. (See Fig. 2.)
WBS Does/Doesn't
There are some things a WBS does not do. For example, it doesn't:

• Clearly identify task dependencies
• Substitute for project follow-through
• Identify the project organization

The WBS does:

• Support scope identification/clarification
• Serve as a baseline for schedule generation
• Facilitate duration estimation
• Allow EVM techniques
• Facilitate responsibility and accountabilities

WBS and Recyclability
Organizations often deliver a series of similar projects (embedded, software, and so on), and they can reuse WBSs from previous projects as a starting point for the new project. On the downside, there is a risk of recycling earlier problems and not accounting for changes since the last use of the WBS. If the organization has conducted a good post-mortem on previous WBS documents, then it should have captured the major issues already and recorded permanent corrective actions.
A project conducted without task details runs the risk of basing its success largely on luck (the use of the "hope" method of project management). A WBS prevents these details from falling through the cracks by accounting for everything that must be done. Our next article will detail how a WBS can be used in scrum project management.

MIL-HDBK-881 recommends the following items for a WBS: Integration, assembly, test, and checkout efforts Systems engineering and program management Training: Equipment Services Facilities Data: Technical publications Engineering data Management data Support data Data depository System test and evaluation: Development test and evaluation Operational test and evaluation Mockups Test and evaluation support Test facilities Peculiar support equipment (items not currently in inventory that must be developed): Test and measurement equipment Support and handling equipment Common support equipment (items currently in inventory): Test and measurement equipment Support and handling equipment Operational and site activation: System assembly, installation, and activation Checkout on site Contractor technical support Site construction Site/ship/vehicle conversion (obviously military and defined as what must be done to accommodate the product; on the civilian side, we would look for opportunities for reuse) Industrial facilities: Construction/conversion/expansion Equipment acquisition or modernization Maintenance (industrial facilities) Initial spares and repair parts

Kim Pries and Jon Quigley are principals with Value Transformation, LLC, a product development training and cost improvement firm. They have written two books for Taylor and Francis (www.taylorandfrancis.com) to be published in 2010 - one on scrum project management and the other on product testing. Contact them at kim.pries@valuetransform.com and jon.quigley@valuetransform.com, respectively.