Follow

Metrics Planning and Measurement

The primary goal of process consulting is a successful SD Elements deployment. It's important to first understand the organizations primary objectives in using SD Elements. These objectives should be tied to well-defined, measurable metrics.

Below we list many of the most common objectives of using SD Elements. You should select the top three that apply to your organization. This list not exhaustive. If your organization has other objectives you should feel free to use them instead of / in addition to this list.

  • Lower application security risk: Application security is a key risk area for the organization and SD Elements is a solution to help reduce it.
  • Lower cost of application security remediation: SD Elements is a mechanism to specify security requirements up-front, thereby reducing the need to fix vulnerabilities after they’ve been coded.
  • Improve developer security awareness: SD Elements is an in-context education system to increase developer understanding of security issues that are relevant to their current work.
  • Improve compliance: SD Elements will be used to improve compliance to and auditability of a particular framework/regulation. For example, to satisfy PCI DSS Section 6.3.
  • Understand application risks: SD Elements is a way to understand the types of risks posed by applications, including risks that existing assessment techniques may be unable to uncover. In addition, it's unclear which risks  existing testing techniques are able to uncover.
  • Disseminate guidance across the organization: SD Elements a system to centralize and automatically push security and other non-functional requirements and guidance in a consistent way across the organization.
  • Scale the security architect role: SD Elements is a system to scale the organization’s existing security architect or other role to cover hundreds/thousands of applications.
  • Increase security of 3rd party developed software:  SD Elements generates detailed technical security requirements for 3rd party software suppliers to comply with.

Metrics for each objective listed above are presented in a tabular form below.

 

NOTE: The list above is not set in stone. If all of an organization’s objectives of using SD Elements are not covered above, add to this list and use as the foundation for defining relevant metrics.

 

Goal

Metric

Description                          

 Common Data Stores

Lower application security risk

Average # of vulnerabilities detected in applications after development

Measure the # of vulnerabilities identified by current security testing process before using SD Elements. Measure again after SD Elements recommendations have been embedded into development.

Note: this may take a long time to measure as it's dependent on the length of time it takes for a development team to completely finish writing and testing code. You should only make use of this metric for long term usage, it is not suitable for short-term / pilot evaluations. 

Security assessment reports

GRC platforms (e.g. Archer)

Enterprise portals for scanners (e.g. Fortify Software Security Center) 

 

Lower cost of application security remediation

Estimated man hours spent on security remediation

Estimated opportunity cost on fixing security vulnerabilities vs. building features or fixing other defects

Interview project managers or application owners in a project where security defects were discovered and had to be fixed. Estimate amount of time spent on fixing security defects. Estimate value to organization of features that were not shipped as a result of fixing security defects.

Do this both before and after using SD Elements.

Note: this may take a long time to measure as it's dependent on the length of time it takes for a development team to completely finish writing and testing code. You should only make use of this metric for long term usage, it is not suitable for short-term / pilot evaluations. 

ALM tools (e.g. JIRA)

 

Project management tools (e.g. CA Clarity, MS Project)

 

Improve developer security awareness

# of previously unknown tasks per application

After modeling an application in SD Elements, measure the number of tasks that were not already implemented / were not already explicitly accepted risks.

SD Elements 

Improve compliance     

% of SD Elements applications in compliance

 

Measure or estimate the % of applications that you have reasonable evidence to believe are in compliance with the relevant sections of a compliance framework / legislation. Observe the difference between those that use SD Elements and those that do not.

Software asset management tools

GRC platforms (e.g. Archer)

SD Elements 

Understand application risks

Qualitative scoring of organization’s ability to understand application risks

Ask security team to describe current state of ability to understand application security risks before and after using SD Elements.  How effective has SD Elements been in giving security teams the ability to understand risks? Measure on a scale of 1-10, where 1 means “No understanding of application risks” and 10 means “Understand all potential application security risks”

 N/A

Disseminate guidance across the organization

Qualitative scoring of organization’s ability to disseminate actionable security guidance

Ask security team to describe current state of ability to distribute security guidance before and after using SD Elements.  How effective has SD Elements been in distributing actionable security guidance? Measure on a scale of 1-10, where 1 means “No ability to distribute security guidance or for development teams to use that guidance” and 10 means “Able to push guidance to all development teams and all development teams can use the guidance”

 N/A

Scale the security architect  role

Average # of applications effectively served per security architect

Ask security architects to measure the # of application teams they can provide detailed application security requirements to before and after using SD Elements.

 N/A

Increase security of 3rd party developed software

Qualitative scoring of organization’s ability to understand application risks in 3rd party software

Number of vulnerabilities detected in applications after procurement

Ask security team to describe current state of ability to measure 3rd party application security risks before and after using SD Elements

Measure the # of vulnerabilities found by security testing discovered in 3rd party applications before using SD Elements. Measure again after using SD Elements.

Security assessment reports

GRC platforms (e.g. Archer)

Enterprise portals for scanners (e.g. Fortify Software Security Center) 

 

 
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments