Performance Measurement

Measuring performance is the key for driving dramatic improvements in any organization.  Choosing the right performance measures, or metrics, is essential to hone the focus of improvement project teams.

The key to honing the focus for the project team is to select measurements that are relevant to the level of the organization that is being impacted by the improvement project.  If the team is working on improvements at the work cell, or production line level within the organization having measurements that resonate with the Corporate Level of the organization won’t make much sense.  The measures must be selected that are appropriate for the level of the organization where the improvements are being made.

Examples of performance measures and the level within the organization follow:

Corporate Level

• Return on Net Assets
• Market Share
• Total Stockholder Return
• Capacity Utilization
• Economic Value Added

Division Level

• Customer Loyalty
• Target Cost Attainment
• Market Share by Division
• Order Fulfillment Cycle Time

Business Unit Level

• Business Unit Revenue
• Business Unit Margin
• Quality Cost
• Process Efficiency
• Staffing Plan Attainment

Work Cell / Production Line

• Cycle Time
• Process Yield
• Employee Satisfaction
• Outgoing Quality

A rating scale can be used to help with selecting the Key Performance Metric, or the CTQ (Critical to Quality) Metric, that will drive the improvement at the given level within the organization.  The rating scale follows:

Rating Scale for Performance Measures

Relevance

1. Not at all linked to strategic objectives
2. Poorly linked to strategic objectives
3. Indirectly linked to strategic objectives
4. Strongly linked to strategic objectives
5. Directly linked to strategic objectives

Usefulness

1. Too detailed to provide useful information
2. Rarely provides useful information
3. Occasionally provides useful information
4. Usually provides useful information
5. Constantly provided useful information

Understandability

1. Very complex, hard to understand
2. Understandable with study
3. Neutral
4. Fairly easy to understand
5. Very easy to understand

Availability of Data

1. Would be very difficult to obtain
2. Will have to be measured manually
3. Can be obtained by combining information on different reports
4. Can be easily derived from information on existing reports
5. Currently available from existing reports

Overall Average Score

1. Extremely poor indicator / motivator
2. Poor indicator / motivator
3. Average indicator / motivator
4. Good indicator / motivator
5. Excellent indicator / motivator

The metric with the highest overall score should be used to drive, monitor, and maintain the gains form the improvement efforts.  Choosing the right performance measures is the key for driving dramatic improvements in any organization.

SMED for Setup Reduction

SMED is an acronym that stands for Single Minute Exchange of Die.  The systematic approach to reaching this nirvana was developed by Shigeo Shingo in the spring of 1950 at Toyo Kogyo’s Mazda plant in Hiroshima, Japan.  He defined a new way of looking at the structure of production by focusing on the relationship between processes and operations.  Here is how to do it.

Shingo defined Processes as:

• Work
• Inspection
• Transportation
• Storage

Shingo defined Operations as:

• Setup Operations
  • Preparation
  • After Adjustments
• Principal Operations
  • Main Tasks
  • Incidental Tasks
• Marginal Allowances
  • Fatigue
  • Hygiene
  • Operations
  • Workplace

SMED for Setup Reduction is comprised of four phases.

Phase 1

This is the as-is, or current state, situation for the production process where both internal and external setup operations are considered to be the same.  There is no separation.  To understand how to make the distinction between internal and external setup operations requires the capture of baseline information and data.

To get the data and information you can:

• Conduct work sampling,
• Conduct interviews with the process participants,
• but the best method is to videotape, or digitally record an entire setup from start to finish.

Phase 2

Now that the baseline has been established the task is to identify and separate the internal and external setup operations.

Definitions:

Internal Setup Operations

• What operations for setup must be performed when the equipment or process has to be idle?

External Setup Operations

• What are the operations for setup that can be performed while current production is taking place before the equipment or process must be shut down and thus idle?

The key to achieving SMED is the ability to distinguish between internal and external setup.

Phase 3

Now that the setup operations have been classified as either internal or external operations it is time to convert internal to external aggressively.  Baseline setup times can be reduced by 30% to 50% or more and this is just the beginning.  Go for it and get even more reductions!

How?

• Make all the conversions identified in Phase 2.
• Was anything wrongly classified as internal?
• Can any of the internal operations be converted to external?
• What do you mean it can’t be done?  Go for it!

Phase 4

Streamline all aspects of both internal and external setup operations.   In this phase look at each of the operations for setup and look for ways to improve them and reduce the time required to get them done.  Nirvana is Single Minute Exchange of Die.  You might not get there that that is the goal!

Phases 3 and 4 can be worked concurrently.  Shigeo Shingo developed this method, SMED, over a period of nineteen years through examining the theoretical and practical aspects of setup improvement.  He developed the cookbook.  It is up to you to apply it.

Keys to Successful Operations Improvement

Operations improvement is not magic, but strategically focused hard work.  The keys to successful operations improvement include these five elements:

• Performance Metrics tied to business goals and objectives
• Leadership agreement on, and support for, select improvement projects
• Project teams staffed for success
• Project centric training
• Accountability for timely successful project completion

Include these elements sequentially in the operations improvement plan as your strategy for success.

Performance Metrics tied to business goals and objectives

The leadership team reviews the goals and objectives for the business and selects the performance metrics that measure success.  Often, Quality Function Deployment (QFD) or a Prioritization Matrix is used to align and rank the performance metrics with the business goals and objectives.  The marching orders are now set for the improvement projects.

Leadership agreement on, and support for, select improvement projects

The leadership team reviews the processes across the supply chain from the beginning to the end.  Where are the issues?  Where do we fall short of expectations?  Where is the waste in the system?  These questions drive the brainstorming to identify potential improvement projects.  A list is made of the projects and incorporated into another QFD or Prioritization Matrix to align and rank the improvement projects with the rated and ranked performance metrics.  Using the matrices keeps the prioritization of the projects aligned with the business goals and objectives.

Project teams staffed for success

The leadership team focuses on the top four or five projects based upon the priorities they have agreed on using prioritization matrices.  The teams for these projects are selected from the best and the brightest personnel within the organization that have knowledge or skills relevant to the project.  Often these individuals are considered too important or too busy to work on improvements.  That is the reason for their selections.  These personnel decisions answer the question, “Are you serious about making improvements?”

Project centric training

Each project and its assigned team progress through the Lean Six Sigma training and apply the tools and techniques directly on their improvement opportunity.  As quickly as methods are learned they are applied to make progress toward the completion of the project following the train and do philosophy.  If the opportunity is a process improvement the methodology follows DMAIC (define, measure, analyze, improve, and control).  If the opportunity is a design improvement the methodology follows DMADV (define, measure, analyze, design, and validate), or Design for Six Sigma.

Accountability for timely successful project completion

Success requires the leadership team and the improvement project teams be held accountable to complete the projects in a timely manner.  Deliverables based project plans focus the effort of the teams to move through the DMAIC, or DMADV, process phases successfully.  Phase exit reviews are held between the project and the leadership teams.  Phase exit reviews assure that all parties involved are held accountable to achieve successful strategically focused operations improvement.

Where do we apply Statistical Tools?

Before starting any type of analysis classify the data set as either continuous or attribute, and in many cases it is a blend of both types.  Continuous data is characterized by variables that can be measured on a continuous scale such as time, temperature, strength, or monetary value.  A test is to divide the value in half and see if it still makes sense.

Attribute, or discrete, data can be associated with a defined grouping and then counted.  Examples are classifications of good and bad, location, vendors’ materials, product or process types, and scales of satisfaction such as poor, fair, good, and excellent.  Once an item is classified it can be counted and the frequency of occurrence can be determined.

The next determination to make is whether the data is an input variable or an output variable.  Output variables are often called the CTQs (critical to quality characteristics) or performance measures.  Input variables are what drive the resultant outcomes.  We generally characterize a product, process, or service delivery outcome (the Y) by some function of the input variables X₁,X₂,X₃,…X_n.  The Y’s are driven by the X’s.

The Y outcomes can be either continuous or discrete data.  Examples of continuous Y’s are cycle time, cost, and productivity.  Examples of discrete Y’s are delivery performance (late or on time), invoice accuracy (accurate, not accurate), and application errors (wrong address, misspelled name, missing age, etc.).

The X inputs can also be either continuous or discrete.  Examples of continuous X’s are temperature, pressure, speed, and volume.  Examples of discrete X’s are process (intake, examination, treatment, and discharge), product type (A, B, C, and D), and vendor material (A, B, C, and D).

Another set of X inputs to always consider are the stratification factors.  These are variables that may influence the product, process, or service delivery performance and should not be overlooked.  If we capture this information during data collection we can study it to determine if it makes a difference or not.  Examples are time of day, day of the week, month of the year, season, location, region, or shift.

Now that the inputs can be sorted from the outputs and the data can be classified as either continuous or discrete the selection of the statistical tool to apply boils down to answering the question, “What is it that we want to know?”  The following is a list of common questions and we’ll address each one separately.

• What is the baseline performance?
• Did the adjustments made to the process, product, or service delivery make a difference?
• Are there any relationships between the multiple input X’s and the output Y’s?  If there are relationships do they make a significant difference?

That’s enough questions to be statistically dangerous so let’s begin by tackling them one at a time.

What is baseline performance?

• Continuous Data

Plot the data in a time based sequence using an X-MR (individuals and moving range control charts) or subgroup the data using an Xbar-R (averages and range control charts).  The centerline of the chart provides an estimate of the average of the data overtime, thus establishing the baseline.  The MR or R charts provide estimates of the variation over time and establish the upper and lower 3 standard deviation control limits for the X or Xbar charts.  Create a Histogram of the data to view a graphic representation of the distribution of the data, test it for normality (p-value should be much greater than 0.05), and compare it to specifications to assess capability.

Minitab Statistical Software Tools are Variables Control Charts, Histograms, Graphical Summary, Normality Test, and Capability Study between and within.

• Discrete Data

Plot the data in a time based sequence using a P Chart (percent defective chart), C Chart (count of defects chart), nP Chart (Sample n times percent defective chart), or a U Chart (defectives per unit chart).  The centerline provides the baseline average performance.  The upper and lower control limits estimate 3 standard deviations of performance above and below the average, which accounts for 99.73% of all expected activity over time.  You will have an estimate of the worst and best case scenarios before any improvements are administered.  Create a Pareto Chart to view a distribution of the categories and their frequencies of occurrence.  If the control charts exhibit only normal natural patterns of variation over time (only common cause variation, no special causes) the centerline, or average value, establishes the capability.

Minitab Statistical Software Tools are Attributes Control Charts and Pareto Analysis.

Did the adjustments made to the process, product, or service delivery make a difference?

• Discrete X – Continuous Y

To test if two group averages (5W-30 vs. Synthetic Oil) impact gas mileage, use a T-Test.  If there are potential environmental concerns that may influence the test results use a Paired T-Test.  Plot the results on a Boxplot and evaluate the T statistics with the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true).  If there is a difference choose the group with the best overall average to meet the goal.

To test if two or more group averages (5W-30, 5W-40, 10W-30, 10W-40, or Synthetic) impact gas mileage use ANOVA (analysis of variance).  Randomize the order of the testing to minimize any time dependent environmental influences on the test results.  Plot the results on a Boxplot or Histogram and evaluate the F statistics with the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true).  If there is a difference choose the group with the best overall average to meet the goal.

In either of the above cases to test to see if there is a difference in the variation caused by the inputs as they impact the output use a Test for Equal Variances (homogeneity of variance).  Use the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true).  If there is a difference choose the group with the lowest standard deviation.

Minitab Statistical Software Tools are 2 Sample T-Test, Paired T-Test, ANOVA, and Test for Equal Variances, Boxplot, Histogram, and Graphical Summary.

• Continuous X – Continuous Y

Plot the input X versus the output Y using a Scatter Plot or if there are multiple input X variables use a Matrix Plot.  The plot provides a graphical representation of the relationship between the variables.  If it appears that a relationship may exist, between one or more of the X input variables and the output Y variable, conduct a Linear Regression of one input X versus one output Y.  Repeat as necessary for each X – Y relationship.

The Linear Regression Model provides an R² statistic, an F statistic, and the p-value.  To be significant for a single X-Y relationship the R² should be greater than 0.36 (36% of the variation in the output Y is explained by the observed changes in the input X), the F should be much greater than 1, and the p-value should be 0.05 or less.

Minitab Statistical Software Tools are Scatter Plot, Matrix Plot, and Fitted Line Plot.

• Discrete X – Discrete Y

In this type of analysis categories, or groups, are compared to other categories, or groups.  For example, “Which cruise line had the highest customer satisfaction?” The discrete X variables are (RCI, Carnival, and Princess Cruise Lines).  The discrete Y variables are the frequency of responses from passengers on their satisfaction surveys by category (poor, fair, good, very good, and excellent) that relate to their vacation experience.

Conduct a cross tab table analysis, or Chi Square analysis, to evaluate if there were differences in levels of satisfaction by passengers based upon the cruise line they vacationed on.  Percentages are used for the evaluation and the Chi Square analysis provides a p-value to further quantify whether or not the differences are significant. The overall p-value associated with the Chi Square analysis should be 0.05 or less.  The variables that have the largest contribution to the Chi Square statistic drive the observed differences.

Minitab Statistical Software Tools are Table Analysis, Matrix Analysis, and Chi Square Analysis.

• Continuous X – Discrete Y

Does the cost per gallon of fuel influence consumer satisfaction?  The continuous X is the cost per gallon of fuel.  The discrete Y is the consumer satisfaction rating (unhappy, indifferent, or happy).  Plot the data using Dot Plots stratified on Y.  The statistical method is a Logistic Regression.  Once again the p-values are used to validate that a significant difference either exists, or it doesn’t.  P-values that are 0.05 or less mean that we have at least a 95% confidence that a significant difference exists.  Use the most frequently occurring ratings to make your determination.

Minitab Statistical Software Tools are Dot Plots stratified on Y and Logistic Regression Analysis.

Are there any relationships between the multiple input X’s and the output Y’s?  If there are relationships do they make a difference?

• Continuous X – Continuous Y

The graphical analysis is a Matrix Scatter Plot where multiple input X’s can be evaluated against the output Y characteristic.  The statistical analysis method is multiple regression.  Evaluate the scatter plots to look for relationships between the X input variables and the output Y.  Also, look for multicolinearity where one input X variable is correlated with another input X variable.  This is analogous to double dipping so we identify those conflicting inputs and systematically remove them from the model.

Multiple regression is a powerful tool, but requires proceeding with caution.  Run the model with all variables included then review the T statistics (T absolute value <=1 is not significant) and F statistics (F <=1 is not significant) to identify the first set of insignificant variables to remove from the model.  During the second iteration of the regression model turn on the variance inflation factors, or VIFs, which are used to quantify potential multicolinearity issues (VIFs <5 are OK, VIFs > 5 to 10 are issues).  Review the Matrix Plot to identify X’s related to other X’s.  Remove the variables with the high VIFs and the largest p-values, but only remove one of the related X variables within a questionable pair. Review the remaining p-values and remove variables with large p-values >>0.05 from the model.  Don’t be surprised if this process requires a few more iterations.

When the multiple regression model is finalized all VIFs will be less than 5 and all p-values will be less than 0.05.  The R² value should be 90% or greater.  This is a significant model and the regression equation can now be used for making predictions as long as we keep the input variables within the min and max range values that were used to create the model.

Minitab Statistical Software Tools are Regression Analysis, Step Wise Regression Analysis, Scatter Plots, Matrix Plots, Fitted Line Plots, Graphical Summary, and Histograms.

• Discrete X  and Continuous X – Continuous Y

This situation requires the use of designed experiments.  Discrete and continuous X’s can be used as the input variables, but the settings for them are predetermined in the design of the experiment.  The analysis method is ANOVA which was previously mentioned.

Here is an example.  The goal is to reduce the number of unpopped kernels of popping corn in a bag of popped pop corn (the output Y).  Discrete X’s could be the brand of popping corn, type of oil, and shape of the popping vessel.  Continuous X’s could be amount of oil, amount of popping corn, cooking time, and cooking temperature.  Specific settings for each of the input X’s are selected and incorporated into the statistical experiment.

Minitab Statistical Software Tools are DOE, Factorial Plots, Pareto Effect Plots, ANOVA, Histograms, and Response Optimizer.

You are now ready to tackle some data, answer some questions, and become statistically dangerous.

Six Sigma and the Bottom Line

Many companies have gone down the path of continuous improvement only to be discouraged by the lack of “breakthrough” results. All of the texts on Total Quality harp on the need for strong commitment from senior management for these initiatives to be successful.  What is it that motivates these business leaders?  The answer is straightforward; Business Leaders are motivated and driven to achieve bottom line results and increase value to shareholders.  Six Sigma provides a structured and rigorous approach with a customer focus that drives benefits to the bottom line.

What is Six Sigma?

• A Program that follows a Structured and Rigorous Approach to Process Improvement
  • Production Processes
  • Service Processes
  • Basically, applicable to all Business Processes

• Where Projects are Identified and Prioritized Based upon “Bang for the Buck”
  • Outwardly Customer Focused
  • Bottom Line Impact Potential is the Key Driver
  • Facilitates allocation of a business’s “scarce” resources to the projects with significant business importance
  • Separates the “Vital Few” from the “Trivial Many”

• Where Specific Targets for Improvement are Base lined and Monitored
  • Cost (hard, soft, and cash flow)
  • Cycle Time
  • Non-Value Adding Activities
  • Rework
  • Failures
  • Defects

The Structured and Rigorous Improvement Process is called DMAIC and is comprised of the following five phases.

• Define
  • In this first phase the projects purpose and scope are defined as well as the initial pass at a business case.  Process and Customer information is collected to identify how well the process is meeting customer requirements.
• Measure
  • The goals of the Measure phase are to establish baseline process performance, narrow the focus of the project scope and problem statement, and increase the accuracy of the business case.  The output from this phase provides the data necessary for the analyze phase.
• Analyze
  • The goals of the Analyze phase are to identify potential root causes of process issues that directly effect critical to quality customer requirements.  Theories are tested and validated with data.  The output from this phase is verified causes that lead to solution development in the next phase.

• Improve
  • The goals of the Improve phase are to develop and pilot proposed solutions, validate that they address the root causes of the process issues, and achieve or exceed the expected benefits.  Once validated, plans are developed for full-scale implementation of solutions.
• Control
  • The goals of the Control phase are to evaluate the new process performance and compare it to the baseline. The business case is updated with the quantified benefits and booked by finance.  Standards, documentation, and training in the new process are completed.  Process controls are institutionalized to maintain the gains and identify further opportunities for continued improvement.

Teams utilizing the five phases of  DMAIC can deliver breakthrough improvements to business processes.  This is the road to the Six Sigma stretch goal of 3.4 defects per million opportunities.

How Do Six Sigma Efforts Impact the Bottom Line?

• Income Statement Elements affected:
  • Reduces Cost of Goods Sold
  • Increases Gross Margin
  • Reduces Operating Expenses
  • Increases Net Income
  • Positive Impact on Profitability Ratios
  • Increases Return on Sales
  • Increases Return on Investment

• Balance Sheet Elements affected:
  • After Processes are Improved and Stabilized
  • Inventory Reductions Become Possible
  • Impacts Activity and Efficiency Ratios
  • Increases Asset Turnover
  • Increases Inventory Turnover
  • Decreases Inventory on Hand

Reductions of 10%-30% in both Cost of Goods Sold and Operating Expenses are common, with most companies averaging 20% reductions. 

• Consider this example:
  • $200M Sales
  • $100M Cost of Goods Sold
  • $90M Operating Expenses
  • $5M Depreciation and Interest
  • Return on Sales ratio of 2.5%
  • $100M Total Assets
  • Asset Turnover Ratio is 2.0
• Given a conservative reduction of 10% in both COGS and OE from a Six Sigma implementation the Return on Sales increases from 2.5% to 12%.
• Without any balance sheet improvements the Return on Investment Ratio increases from 5% to 24%.
• Inventory is generally addressed after processes have been improved and the need to carry excess inventory to meet customer demand has been reduced.
• A project to reduce the cycle time on the accounts receivable process is often addressed early on in six sigma implementations.

How Do Six Sigma Efforts Do It?

• The key to Six Sigma successes at achieving bottom line improvements is following a Structured and Rigorous Improvement Process that is Customer Focused.

The DMAIC Process comprises the following:

• Define Elements
  • Project Charter and Business Case
  • High Level Process Map called a SI P O C
  • Suppliers to the process
  • Inputs to the process
  • Process steps (5-7 at this level of detail)
  • Outputs from the process steps
  • Customers of the process outputs
  • Voice of the Customer information
  • Needs
  • Drivers
  • CTQs (Critical to Quality Requirements)

• Measure Elements
  • Data Collection Plan
  • Sampling Strategy for Collecting the Data
  • Measurement System Analysis
  • Data Collection
  • Graphically Assessing Patterns in the Data
  • Establishing the Baselines from which to Measure Improvements
  • Business Case Metrics
  • Process Sigma Levels

• Analyze Elements
  • Organize potential root causes using:
  • Affinity Diagrams
  • Cause & Effect Diagrams
  • Process Analysis
  • Analyze Cycle Time
  • Identify Value Added and Non-Value Added activities
  • Data Analysis
  • Identify variation over time
  • Identify relationships between process inputs and outputs
  • Hypothesis Testing (Theory Testing)
  • Does one or more of the potential root causes actually make a difference?
  • Here is where we answer those questions.
  • Regression Analysis of Historical Data
  • Identify the process variables that if controlled, allow us to predict with great certainty the outcome of the process.
  • For example, what is the time (+/- some number of minutes) and the temperature (+/- some number of degrees) that will cook the perfect pizza based upon historical customer feedback of their critical to quality preferences?
  • Design of Experiments
  • Testing that is designed to specifically identify which process variables affect process outputs and is conducted in a controlled fashion.
  • Provides the ability to test many variables where historical data may not have existed, or was not collected in a manner allowing analysis.

• Improve Elements
  • Develop potential solutions using Brainstorming and Advanced Creativity methods.
  • Conduct FMEA (Failure Mode and Effects Analysis) to assess risks and potential failures of the proposed solutions.  Try to understand and determine what could go wrong?
  • Rate the severity of a potential failure.
  • Rate the Occurrence of a potential failure.
  • Rate the ability to detect the existence of the failure.
  • Identify the best solutions to minimize failures, or defects.
  • Test the best possible solutions in a pilot implementation.
  • Using the Analyze phase tools answer the question, “Is there a difference with the new method versus the old method?”
  • Select the solutions that actually make a difference.
  • Plan for full-scale implementation.
  • Implement the solutions on a full-scale basis.

• Control Elements
  • Determine and implement process controls.
  • Develop, Document, and Implement Standards.
  • Procedures
  • Policies
  • Operational Definitions
  • Monitor process performance metrics.
  • Evaluate the improvement from the baseline to the current improved state.
  • Determine and quantify the improvement in terms of cost, cost avoidance, revenue enhancement, cycle time, and customer satisfaction.
  • Close the project and celebrate a job well done by the Six Sigma team.

Six Sigma is not just another “Quality” program like TQM, or Quality Circles, but one that has teeth rooted in the financials of the business.  Projects are only undertaken if they meet strict business case guidelines set by the senior management team.  Not all projects make the cut.  Some typical minimum thresholds for a green belt project are $150k cost or expense reduction, $500k cost avoidance, or $1000k cash flow depending on the nature of the business process.  The first wave of 5 to 7 projects, completed within 12 to 14 weeks after green belt training, should yield $1 to $2 Million in annualized benefits.  That’s bottom line “breakthrough” results.