Testing Complex and Embedded Systems

by Kim H. Pries, Jon M. Quigley

Many enterprises regard system-level testing as the final piece of the development effort, rather than as a tool that should be integrated throughout the development process. As a consequence, test teams often execute critical test plans just before product launch, resulting in much of the corrective work being performed in a rush and at the last minute. Presenting combinatorial approaches for improving test coverage, Testing Complex and Embedded Systems details techniques to help you streamline testing and identify problems before they occur―including turbocharged testing using Six Sigma and exploratory testing methods. Rather than present the continuum of testing for particular products or design attributes, the text focuses on boundary conditions. Examining systems and software testing, it explains how to use simulation and emulation to complement testing. Where you find organizations that are successful at product development, you are likely to find groups that practice disciplined, strategic, and thorough testing. Tapping into the authors’ decades of experience managing test groups in the automotive industry, this book provides the understanding to help ensure your organization joins the likes of these groups.

320 pages, Hardcover

First published December 8, 2010

About the author

Mr. Jon Quigley works for Volvo 3P as the manager of the Electrical and Electronics Verification and Test group in North America. He is also a principal in Value Transformation, LLC--a consultancy providing value engineering, project management, test engineering, and training primarily in new product development.

Jon has experience as an embedded product developer (hardware and software) for industrial applications, as well for the heavy vehicle industry. He has had roles as a test engineer for projects, both for automotive and heavy vehicle applications. He is experienced in software engineering, software testing, embedded product engineering, test engineering (production and product testing), book writing, Product Development training, automotive quality, and real-time programming.

Jon is the coauthor, with Kim H. Pries, of Project Management of Complex and Embedded Systems (CRC Press, 2008), as well as more then a dozen magazine and e-magazine articles. These articles are in Automotive Design Line, EDN, Embedded Systems Design, Product Design & Development, Software Magazine, Software Development Times, Software Test and Performance, Quality Magazine, and Project Magazine on line. Additionally, Jon is co-authoring two other books with Kim Pries that are under contract. The books are tentatively titled: "Scrum Project Management" and "Testing of Complex and Embedded Systems" both will be available middle to late 2010.

Jon has a bachelor's degree in electronics engineering from the University of North Carolina at Charlotte, and masters of business administration in marketing as well as a masters of science in project management from City University of Seattle. He has five US patents, with another three in various stages of the patent process. Additionally, he was part of the team that received the North American Volvo Technical award in 2005 going on to win the Volvo Global Technology Award in 2006. He is a member of SAE, SAVE, and PMI, and is the holder of Project Management Professional (PMP) certification. He lives in Lexington, North Carolina with his wife Nancy, and son Jackson a cat and a dog."

Reviews

[Chris Maddix · Rating 5 out of 5]

Let's start out with what this book is not. It isn't lighthearted, it isn't romantic, there is no plot, no climax, no action hero, no happy ending. It also isn't for the faint of heart. Are you comfortable with the terms abstraction, verisimilitude, stochastic, exogenous, etc.? If not, then leave this book for the professionals. But, more importantly, what is this book? It is a how-to manual for ensuring that quality products are developed. It is written in a generic fashion - I can't offhand think of a product for which this description of testing would not work. It's also greatly clarified and in places simplified by a plethora of charts, graphs, drawings and schematics. If I were running a company, I would hope to hire individuals well-versed in the procedures described herein; if I lacked full confidence in their ability to test my product correctly, I might just refer them to this book.

Here is a sampler of some of the aspects of testing discussed:

- testing should come hand in hand with product development, rather than waiting for the period just prior to introducing the product to the customer
- practice does not make perfect; perfect practice makes perfect; improved testing methods will enhance product quality and reliability
- the goals of testing are to discover, prevent and contain defects; it shouldn't be left to the customer to discover defects
- testing should be guided by the question "What's wrong with it?" rather than "Why is it good?"
- there are literally over a dozen parameters to test, including environmental factors, repeatability, and uncontrollable factors, which the authors refer to as "noise"
- testing requires planning and expected documentation
- there are "bad" and uncontrollable tests
- testing should factor in uncertainty, such as unreliable measurement, test bias, evaluations of "pass" or "fail", and statistical validity

My thanks to the authors and to the Taylor & Francis Group for producing this excellent manual.

[Filippe · Rating 5 out of 5]

Testing Complex and Embedded Systems is an excellent book. If you're seeking a comprehensive source of testing information, this book is a must-read. It covers a wide array of subjects, including software, mechanical, electrical, metallurgical, industrial, manufacturing, and more. Many of the insights presented in the book are not commonly utilized in most organizations. Therefore, by absorbing the information within this book, you can gain a significant advantage over others in these areas.

One noteworthy aspect of this book is its use of analogies from the field of philosophy, making it easier to comprehend complex matters. While a solid understanding of economics and math may be beneficial for certain sections, most of the content doesn't require extensive familiarity with complex subjects.

The book proves to be immensely helpful for various technology-focused companies. It not only guides organizations in improving product quality but also offers insights on enhancing consumer satisfaction. Reading this book can illuminate why a product may not be performing as expected.

Though the book could benefit from more analogies and images, I find it to be a valuable resource. It provides a wealth of information, offering a comprehensive understanding of testing across different disciplines. With this book, you can address questions like when is the best time to initiate testing in software development.

In summary, Testing Complex and Embedded Systems is an indispensable guide. While it could use some improvements in terms of didactics and visuals, it remains a worthwhile read for anyone interested in gaining a profound insight into testing.

[Boyko Ovcharov · Rating 5 out of 5]

Well researched, informative, derived from practical experience and coupled with relevant theories, models and paradigms.

It utilizes a multidisciplinary, critical approach to testing, quality assurance, learning and improvement, product/service development etc. across various major industries, not only in the field of IT systems.

Intended for executive managers who are responsible for strategic planning in an organization.
The complex processes this book deals with are crucial for the success of an enterprise, a product or service launch, including customer satisfaction.

People and systems need to effectively work together in order to achieve the desired results. Things are interconnected and require close attention.

In conclusion, the book sounds serious, while raising important questions for businesses today.

Recommended to industry professionals.

[Syed Naqvi · Rating 4 out of 5]

Testing is one crucial part of any product development especially when it concerns critical embedded parts (MCU's, chips or other electronic components). This book covers the embedded systems testing especially related to automotive segment. Developing suitable test plans ensures that the product will fulfil its intended purpose without unknown failure modes (exceptions do exist however as it is practically not possible to test / check for all possible combination of factors).

One excellent way to do advanced and system level testing is by use of Simulators. Live testing is not always possible especially with various combinations of complicated changing factors.
A very knowledgeful read especially for those interested in embedded products development, design, coding etc..

[Pasang · Rating 5 out of 5]

This book is suitable for those who holds leadership position in the organization and wants to ensure quality in its activity.It has detailed explanation on how an organization can capture the learning and use it in improving the performance.It has given holistic and generic explanation on learning and embedding it in the organizational activities.Anyone who wants to understand how the organization or project can perform quality activity and operate successfully,but do not hold managerial knowledge can be benefitted with this book.And those who already have knowledge can broaden the understanding.

[Het · Rating 4 out of 5]

Sending out a fine quality product is great for the customer as well as the company as it will not only improve their name but also save on refunds and repairs. Testing is very integral to the production process. The book not only succeeds to emphasis that but also gives the background or basic enough with relevant solutions or processes that can be used. It shows the flip side as well.
Overall I liked reading it as the content was presented in a systemic manner with images, graphs, diagrams, flow charts and tables.

[Kelly · Rating 4 out of 5]

I liked the formatting and general design of this book. There was a lot of information there, and it was quite wordy, but the text was broken up with striking diagrams to help illustrate the points. The detailed information was presented under clear sections and headings, and the individual chapters were fairly short. Maybe this is not a book that would always be read cover to cover, but instead may be referred to when required. Either way, I think it could be quite useful.

[Swati · Rating 5 out of 5]

This book is amazing in my way... I don't deserve this book because I couldn't understand much parts because of my interest but I think if I know technology things and diagram, I may loved it... And I must share it with my friends who have intrest in this genres.... Please give a chance to this book if you're interested in enterprise, development things...

[Naya Gamal · Rating 4 out of 5]

This book is straight to the point and not for everyone. It’s dense, technical, and definitely not casual reading. But if you’re serious about testing and quality, it’s worth it. The focus on finding flaws instead of praising what works really stood out to me. It’s practical, detailed, and useful across industries. A tough read, but super valuable.

[Jesse Nyokabi · Rating 5 out of 5]

Many enterprises regard system-level testing as the final piece of the development effort, rather than as a tool that should be integrated throughout the development process.

Therefore, test teams often execute critical test plans just before product launch, resulting in much of the corrective work being performed in a rush and at the last minute.

Presenting combinatorial approaches for improving test coverage, Testing Complex and Embedded Systems details techniques to help you streamline testing and identify problems before they occur―including turbocharged testing using Six Sigma and exploratory testing methods.

Rather than present the continuum of testing for products or design attributes, the text focuses on boundary conditions. Examining systems and software testing, it explains how to use simulation and emulation to complement testing. Where you find organizations that are successful at product development, you are likely to find groups that practice disciplined, strategic, and thorough testing.

Tapping into the authors’ decades of experience managing test groups in the automotive industry, this book provides the understanding to help ensure your organization joins the likes of these groups.

Some enterprises regard testing as the final piece of the development effort rather than as a competitive tool. Consequently, the test team executes critical test plans as the project is closing—just before launch and well after the time when the design and development teams can perform any kind of rational corrective actions.

This behavior affords little time for the test engineers to understand the product at a sufficiently detailed level to perform useful testing. Testing groups may adopt a fatalistic approach to their craft by realizing that dilatory sample delivery and schedule crashing is part of their destiny or possibly the result of truly incompetent planning. The situation is aggravated by disengagement between the development and verification groups, as well as a frequent disconnect between project management, development engineering, and test engineering.

The seriousness of this situation is illustrated when you hear project managers lament the test group “blowing” the schedule when they refuse the request to test the system before the constituent system components are even available for test.

We owe it to our customers to provide them with high-quality, reasonably priced, on-schedule, and safe products. Test engineering is a huge driver for achieving this goal because it is through testing that we reveal the character of our product. If we are professional enough and careful enough, we can cautiously predict the general quality of the product we are about to sell. Intelligent product testing should eliminate surprises in the field.

Improved reliability has the obvious benefit of lowering the maintenance cost of the product—return costs will be less, as well as the handling of these returned parts. Fewer failures mean fewer failed parts to manage, handle, and find the root cause and corrective actions. These activities can take a lot of time and resources that do not add value to the supply chain but drag it down.

Ultimately, customers do not want failure. It will not matter whether the product fails due to design, manufacturing, or unanticipated use of the product. Happy customers are ones whose product experiences are not complicated by failures or unpredictable performance, anomalies, and the hassle of returning the product.

The objectives of testing, at a minimum, are to make sure the product meets the customer’s expectations. This is a minimal approach—to be successful, the goal of testing is to
■ Discover product defects
■ Prevent defects
■ Contain defects to a single release
■ Analyze for statistical release readiness
■ Discover product failure limits

The verification and test group is there to provide a critical and unbiased review of the product. This is used to understand the real quality of the product and adjust and improve that quality. When we find a bug or defect, we can consider whether or not it gets corrected before the product is shipped to the customer. Without this work, the first opportunity to ascertain the product quality would be the customer.

My thanks to the authors and the Taylor & Francis Group for producing this excellent manual.

Being an Engineer, I recommend this book.