What To and Not To Test – Mark Seemann

Blog Post:


The Purpose of Testing

  • To exercise the Api and the design. A hard class to test, is a poorly designed interface.
  • To prevent regressions.

The Cost of Regressions

Dimensions of the Risk

  • The likelihood the event.
  • The impact of the event.

To reduce risk, you either decrease the likelihood or the impact of the event.

What is the impact of the error if happens? If it’s a dev app, there is almost zero risk. If it’s a Voyager probes. It could doom the project.

Udemy – TDD Class


  • Fast
  • Independent
  • Repeatable
  • Self-Validatable
  • In-Time, written before production code

Types of Tests

  • Unit Tests – verify the behavior of a unit under tests in isolation
  • Integration Tests – verify the behavior of either a part of the system or a whole system. These tests are often brittle.
  • Acceptance Tests – verify the software from the user’s point of view.

Frameworks & Tools

  • SpecFlow – BDD
  • xUnit – unit testing
  • Moq – Mocking
  • NCrunch
  • Resharper


  • Triangulation
    • We generalize the implementation when we have two or more test cases. We add test cases until the right way of implementation starts to emerge.
  • Faking
    • Faking means returning constants and gradually replacing the value with variables and evolving the solution.
  • Obvious Implementation
    • Sometimes the implementation appears obvious, but as you start to poke at it with tests, it begin to crumble.
  • Grabbing the gold
    • You start from writing unit tests which cover the core functionality right away, and it fails with edge cases.
    • Suggestion: Write as many tests as possible before approaching the core functionality. Check for max values, empty strings, null values, min values… etc.
    • Uncle Bob’s Suggestions: Write tests exactly in the following order: exceptional, degenerate, and ancillary
      • Degenerative cases are those which cause the core functionality to do "nothing"
      • Ancillary behaviors are those which support the core functionality

The Future of Software Engineering – Mary Poppendieck

One database that is a magic integrator of everything else. In a traditional application (circa 2019), most application store all the data in a central store. For example, user data and application data are stored in the same database.

Moving Beyond a Database

"We need to learn how to do application Architecture with API’s not databases."

  • Think of API’s as architecture, they localize architecture. It’s just me an that other server over there. Not me and those 50 other services.

  • API’s have local persistence.

  • Think how we connected to multiple API’s access stored data. i.e. There isn’t one big database to access data from. We have to piece together the data for our system to work.

Big Data Systems are inherently distributed and their architectures must explicitly handle

  • partial failures
  • concurrency
  • consistency
  • replication
  • communication latenecies

Resilient Architectures

  • Replicate data to ensure availability in the case of failure
  • Design componetes to be
    • stateless
    • replicated
    • tolerant of failures of dependent services

Thinking in Antifragile

  • Things better when it gets attacked
  • Chaos monkey

Change in thinking

  • Stop thinking of serial (sync) execution and start thinking in concurrent (async) execution
  • We need to (re)learn event driven programming, instead of procedure programing

Software Engineering in the Cloud



We poke the system, and then respond to the small contained poke.

Continuous Delivery

  • Deploying to the trunk
    • Having no branching
  • The code is always projection ready
  • Deploying all the time
  • Release is turning something on by a switch (enable and disable features)


End to End feedback

  • Monitor the code and it’s health
  • Get feedback from the customers

Get the product feedback to the stakeholders.

Likely 2/3 of the features are unnecessary.

Move from delivery Teams to problem solving teams.

  • A delivery team is given an order and get things done. Like ordering food at a restaurant
  • A Problem Solving team
    • Here’s the goal, here’s the metric, figure out how to improve the metric.

Experimentation and Learning

Conformity Bias: Those who think they are in the minority self-silence


Book Recommendations

Why Scaling Agile Doesn’t Work – Jez Humble

Water – Scrum – Fall

  • Dave West

Design and product discovery usually is defined upfront, which is not agile, adding an agile development process in the middle isn’t all the benefits, especially with the testing and the deployment not part of the process.

Estimating the amount of work

What Should We do

  • Don’t optimize for the case where we are tight
  • Focus on value, not cost
  • create feedback loops to validate assumptions
  • make it economic to work in small batches – working in small batches allows us to get the process working (i.e. faster feedback loops).
  • enable an experimental approach to product dev
    • Allows developers to test a hypothesis on a feature and get almost immediate feedback.


  • Users don’t know what they want. They know what they don’t want. Requirements are the wants of the HIPPO, the highest-paid person.
    • A lot of the time the value isn’t defined. The "So that…" definition of value is never discussed or defined.

Tool Impact Mapping

A tool that maps the impact of the outcome. We work back from the outcome to the code of adding the feature.

The important thing is the shared understanding of the thing you’re building, not the artifacts, which agile emphasizes. This means the developers understanding the "Why" behind something, instead of tossing tasks over the cubical wall.

Pick a task with the smallest amount of work for the biggest outcome. "Minimize output, maximize outcome" – Jeff Geofels

Hypothesis-driven delivery


Is about delivering things in a way that will satisfy our customers. The hard part is trying to discover what the customers want without sinking a whole ton of money into discovering, because, then you might get sucked into the sunk cost fallacy, where you’ve put a ton of money into a solution, and because of this, it’s the correct approach.

Do Less

Only 1/3 of the features improved the key metric. As Jez pointed out, this means 2/3rd’s of the work one doesn’t bring value. In fact, some of the work brings negative value to the company. This means 2/3rd’s of the time someone could spend time on the beach and have the same effect.

Feature Branching is a poor person modular’s architecture

HP Firmware Case Study

The Case Study: A Practical Approach to Large-Scale Agile Development – Gruver, Young, Fulghum


  1. Get Firmware off the critical path
  2. 10x productivity increase.

The incremental approach doesn’t just apply to software, but how we improve our processes and how we improve our companies. It’s how we improve anything.

The Four Steps of Improvement Kata Model

If you’re not constantly improving, you are degrading.

To get started, start with your job and then start networking with people in the company to understand their pain points.

Don’t fight stupid, make more awesome. – Jessie Owens

Tactical Design Patterns

Apply the design pattern when the design is ready to accept it.

  • Don’t force a design pattern onto a problem. Let the code evolve until it’s obvious the design pattern is needed. It is better to be late than early.

  • This idea comes back to my thoughts on keep things simple don’t over architect code. Don’t add it, until it’s a requirement.

Multiple Iterations

  • The first attempt is to produce the tailored design
    • Design patterns do not fit into this phase
  • Then refactor to reach a better design
    • this is where design patterns fit well.

Principle of Least Surprise

Design Patterns are generally more complex than they are in the examples and text books.

Software Architecture Fundamentals: Understanding the Basics

Lessons Learned as an architect

  1. negotiation and political skills are sometimes, and most often, more important as an architect than technology skill
  2. It’s all about the data
  3. The world’s best architectures are not the perfects ones, but rather the feasible ones.

What is software design?

Our plan, our design is the complete source code.

Neil Ford One of the reasons Software is so complex is we don’t have any constraints like the real world. So we don’t have any limits on what we can do.


"The final goal of any traditional engineering activity is some type of documentation."

"When the design effort is complete, the design documentation is turned over to the manufacturing team"

Fixing a manufacturing defect in from a factory is expensive and costly. Much effort is put into place to eliminate defects before the final goods are created. In software, compiling (manufacturing the software) is so cheap, that it’s cheaper to manufacture (compile) the software and test that the item matches our expectations. This is analogous to planning a pre-manufacture rigor found in the manufacturing industry.

In the software world we are in the world of bits not atoms. This makes making changes over time much easier than in the world of atoms.

The plan in the manufacturing world is the source code in the software world.

"Given that software designs are relatively easy to turn out and essentially free to build, an unsurprising revelation is that software designs tend to be incredibly large and complex." – Jack Reeves

Each piece of software is a handcrafted piece. We don’t have the componentization that the real-world has.

In traditional engineering, plans are engineered up front large amounts of time and money are invested to get it right before it’s implemented. In software, manufacturing is so cheap, that we can create it up front and then test to ensure that it’s implemented correctly.

"Software may be cheap to build, but it is incredibly expensive to design." – Jack Reeves

You want to architect for change. This does not mean you want to have mass abstractions. This means pithiness and keeping is simple. If you can make changing architecture less expensive to do, it gives your flexibly of evolving your application over time.

Software always becomes iterative. You will always need to change or fix the software. The difference is how tight your feedback loop is. In waterfall it might be a long drawn out process, in agile it’s short a short feedback loop.

Architecture also becomes iterative. No one architecture/approach can solve every problem. Understand why things work, not mechanics of what things are doing.

Separate goals from approaches.

Architecture isn’t an equation to be solved; it’s a snapshot of a process. It’s always in flux, it’s always changing, a perfect architecture always fails when it meets the real world.

Architecture is coupled to process (especially, continuous delivery). It is coupled to the technical solution and to the process. This is what makes architecture so difficult.

Course outline The 3 area’s of software architecture

  1. Application
    • Techniques for change
    • patterns & anti-patterns
    • tools & documentations
  2. Integration
    • overview
    • architectural styles
  3. Enterprise
    1. introduction to enterprise architecture

Architecture cross-cutting concerns

  • Soft Skills
  • Continuous Delivery
  • Understanding large codebases

Soft Skills

Expectation of an Architect

  1. Analyze technology, industry, and market trends and keep current with those latest trends.
  2. Analyze the current technology environment and recommend solutions for improvement.
    • Look at the full life-cycle of the project. It is the responsibility of the architect to make sure it works. This includes thinks like process and continuous delivery.
  3. Ensure compliance with the architecture.
    • This is an ongoing responsibility of an architect.
  4. Having expose to multiple and diverse technologies, platforms and environments.
  5. Possess exceptional interpersonal skills, including teamwork, facilitation, and negotiation
  6. define the architecture and design principles to guide technology decisions for the enterprise.
    • Don’t dictate specific technology choices.
  7. understand the political climate of the enterprise and be able to navigate the politics.

Architecture Aspects

  • Leadership and communication
  • technical knowledge
  • business domain knowledge
  • methodology and strategy

Continuous Delivery – "Fast, automated feedback on the production readiness of your application every time there is a change — to code, infrastructure, or configuration" **** Software is always in a production ready state. Any change will trigger a build. This includes software patches on a server or stalling new software on the server.

  • integrate early and often
    • The longer you put it off, the worse it gets at an exponential level.
    • At one point in history, people would go off and create their own bits of software. After a short, or sometimes longtime, they would return and try to integrate the different pieces. More often than, not, the pieces were very difficult, if not impossible to put together. It would often take months to put all the pieces together on a big project.
  • As with most things in software, the longer you put it off, the harder it gets.
  • Continuous Delivery is the final stage of CI.

continuous delivery ideal

  • software is always production ready
  • deployments are reliable and commonplace
  • everyone can self-service deployments
  • releases occur according to business needs, not operational constraints


A change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior.

Any fool can write code that a computer can understand. Good programmers write code that humans can understand. – Martin Fowler

Technical debt is the sum of the following coding shortcuts:

  • Duplication
  • Excess Coupling
  • Quick Fixes
  • Hacks

Why Refactor?

  • Improves Design
  • Improves Readability
  • Reveals Defects
  • Helps You Program Faster

Other than when you are very close to a deadline, however, you should not put off refactoring because you haven’t got time. Experience with several projects has shown that a bout of refactoring results in increased productivity. Not having enough time usually is a sign that you need to do some refactoring. – Martin Fowler

Refactoring Principles

  • Keep it Simple
  • Keep it DRY
  • Make It Expressive
  • Reduce Overall Code
  • Separate Concerns
  • Appropriate Level of Abstraction

Refactor Tips

  • Keep Refactoring’s Small
  • One At a Time
  • Make a Checklist
  • Make a ‘later’ list
  • Check in Frequently
  • Add Test Cases
  • Review the Results

Premature optimization is the root of all software evil. – Donald Knuth

Coding Philosophy

  • Keep it simple.
  • Deliver value.
  • Avoid over-engineering.
  • Avoid gold plating your code.

Principle of Least Surprise

  • Do what users expect
  • Be Simple
  • Be Clear
  • Be Consistent
  • Design API to behave as programmers would expect.

Organizing Code Smells into 5 groups

  • The Bloaters
    • Do it all god objects…
      • Small lean code bases and changed quickly
    • Long Methods
      • Keep methods smaller
        • 10 lines or fewer
        • Avoid regions
  • The Object-Orientation Abusers
  • The Change Preventers
  • The Dispensables
  • The Couplers
  • The obfuscators
  • Environment Smells
  • Test Smells
  • Architecture Smells

Refactoring Long Methods

  • Extract Method
    • Introduce Parameter Object
    • Replace Temp with Query
    • Replace Method with Method Object
  • Compose Method
    • Place bits of functionality into smaller methods.
    • The result is a method that has a number of method calls.
  • Replace Nested Conditional with Guard Clause
  • Replace Conditional Dispatcher(switch statement) with Command Pattern
  • Move Accumulation to Visitor ()
  • Replace Conditional Logic Strategy pattern

Refactor Large Class

Violating Single Responsibility Principle

Too many instance variables

Too many private methods (Iceberg class)

Lack of cohesion

Some methods work with some instance variables, others with others

Compartmentalized class

Refactoring Large Classes Strategy

Extract Method (and hopefully combine logic)

Extract class

Extract Subclass/Extract Interface

Replace Conditional Dispatch with Command

Replace State-Altering Conditionals with State Machine

Replace Implicit Language with Interpreter Pattern

Primitive Obsession

  • Over-use of primitives, instead of better abstractions, results in excess code to shoehorn the type. The primitives themselves can contain invalid states and therefore must be checked when the value is set or used.

    • Often Guard clauses and validation is used to everytime the value changes for the type.
    • Les Intention – Revealing code
  • Refactor away from Primitive Obsession

    • Replace Data value with Object
    • Replace Type Code with Class
    • Replace Type Code with Subclass
    • Extract Class
    • Introduce Parameter Object
    • Replace Array with Object
    • Replace State-Altering Conditionals with State
    • Replace Condition Logic with Strategy
  • Examples of a Primitive Data Obsession

    • ZIP / Postal Codes
    • Phone Numbers
    • Social Security Numbers
    • Telephone Numbers
    • Money
    • Age
    • Temperature
    • Address
    • Credit Card Information

Long Parameter List

  • May indicate procedural rather than OO programming style

  • Related Smells

    • Message Chains
    • MiddleMan
    • Rules of Thumb
      • No more than three arguments
      • No output arguments
        • Not intuitive
      • No flag arguments
        • means the method is doing more than one things
      • Selector arguments
        • MEans the method is doing more than one thing.
  • In general

    • Prefer more, smaller, well-named methods to fewer methods with behavior controlled by parameters

Evolutionary Architecture – Rebecca Parsons

What does good look like? What are characteristics are good, what characteristics are bad?

Design and Architecture – Big ‘A’ Architecture. How do different parts of the system communicate with each other. What is our security? What is our persistence strategy? Design are things that are closer to the code. How the system interacts with devices is Architecture.

What are the principles of Evolutionary Architecture?

  1. Last responsible Moment
    • Delay decisions as long as you can, but no longer
    • Maximizes the information you have
    • Minimizes technical debt from complexity
    • Decide early what your drivers are, and prioritize decisions accordingly
    • Evolutionary, neither emergent nor based on guesswork
  2. Architect and develop for evolvability
    • Sensible breakdown of functionality
    • Consider data lifecycle and ownership
      • Who has access to it
      • How long should we keep it.
    • Appropriate coupling
      • Coupling is inevitable
        • Two systems that must communicate, there has to be coupling.
    • Lightweight tooling and documentation
    • Develop for Evolvability
      • Software internal quality metrics focusing on each of change
        • You don’t want to optimize for understanding code, you want to optimize for understanding code
        • OnceI have these metrics, how to I focus my attention on making a piece of code more evolable.
        • Look for change hotspots
        • Measure continually, focus on trends.
          • Gives visibility into changes and software evolution
    • Reversibility
      • Change that decision later down the road.
  3. Postel’s Law
    • How systems communicate with each other.
    • Be conservative in what you send, if you put it out there people will use it, because you are now committed.
    • Be liberal in what you receive. If you only want a phone number, but receive the entire address. Only validate the phone number. You don’t want to be sensitive to changes that don’t impact you.
    • Use version numbering when you have to break a contract.
  4. Architect for testability
    • If you think about testability in your architecture, the side effect is a well architected system. Things that tend to make your system hard to tests also tend to make it hard to change and maintain.
    • Single Responsibility Principle – Example: Messaging infrastructure used for messaging, not business logic. Testing business logic embedded in Messaging is hard. Separate the two and test business logic outside the message infrastructure.
    • Map your business components to the business concepts. When the business requests changes, in a business concept if your components are similarly mapped the change will be less impactful.
    • Automated build and deployments is an essential underpinning
  5. Conway’s Law
    • Organizations design system reflecting their communication structures
    • Broken communications imply complex integration
    • Silos often result in broken communication
    • If you don’t want your product to look like your organization, change your organization (or your product)

Techniques for implementing evolutionary architecture

  • Database Refactoring
    • Decompose big change into series of small changes
    • Each change is a refactoring/migration pair(or triple of you include access code)
      • This should be easy, but data gets dirty overtime making it difficult.
    • Changes compose in the same way functions compose
    • And of course, version control the changes
    • And apply in the various environments during promotion
  • Continuous Delivery
    • Automate environments and configuration
      • Is the underpinning that makes everything works.
    • Automate builds and deployments and use continuous integration
    • Automate testing at all levels
    • Deployments should be boring!!
    • Just because you CAN release at any time, doesn’t mean you HAVE to
  • Choreography
    • How different parts of the system works together as opposed to orchestration
    • Scripted outcomes and vision
    • Individuals perform to the vision without a conductor
    • Distributes authority about interactions
    • Introduces new kinds of failure scenarios
  • Contract Testing
    • Acceptance tests at the systems interface
    • Documents assumptions made
    • Maximizes parallel independent work
    • Used in congestion with Postel’s Law
    • One traditional role of Enterprise Architect

Evolutionary Architecture

  • Define your architectural fitness function
    • Must think about this upfront
      • What is most important to you
        • What is your security threat landscape?
        • What are your scaling requirements?
        • What are your performance requirements?
        • What are the things you can’t compromise on?
  • Delay your decisions as long as you can
  • Understand various forms of technical debt
  • Implement evidence based re-use.
  • Create and maintain the testing safety net

Book Recommendations

  • Patterns of Enterprise Architecture
  • Enterprise integration PAtterns
  • Craft of being a Architecture
  • Data Architecture

Rebecca Parsons – Publishing a book on

Technical debt is some drag on your development process because of some technical issue.

Uncoupling – Micheal Nygard

Coupling is the difficulty when changing things.

Tensegrity – Holds it self together by the tension among parts and the resistant tension from the beams. It’s a nest of opposing forces.

Coupling is everywhere. We are coupled to the Earth. The Earth is coupled to the Sun. The solar system is coupled to the Milky Way. The same is true if we go to the atomic level.

Coupling defines the degrees of freedom. It constrains and enables some movement. It also connects effects.

Everything is coupled to everything

  • ambient temperature
  • ambient atmosphere
  • acoustic vibrations
  • Electromagnetic field
  • Gravity
  • Higgs field
Type Effect
Operational Consumer cannot run without the provider
Development Changes in producers and consumers must be coordinated
Semantic Change together because of shared concepts
Functional Change together because of shared responsibility
Incidental Change together for no good reason (e.g., breaking API changes.)

A shared concept is an idea that is exposed/implemented in one of area of the system. And then other consumers (external or internal) must embrace the concept to consume the resource. However, if the concept is flawed, that flaw proliferates throughout the system.

System Diagrams – Long Arrows

  • Arrows connecting two systems is actually hiding a number of moving parts
    • Extract tables to files
    • Push files across network
    • Load data into database
    • etc…
  • Architecture Qualities in Long Chains
    • Latency strictly warse than the slowest link in the chain
    • Availability strictly worse than the least available link
    • Throughput strictly worse than the throughput of the worst bottleneck
    • Security strictly worse than the security of the weakest link
  • Digging Out – Information Hiding (Decision Hiding)
    • On the Criteria to be used in Composing Systems into Modules – David Parness

Composing Interfaces

A smaller number of well defined interfaces is preferable over more interfaces and small number of concise datatypes is preferable over more datatypes.


  • Inversely proportional to the # of interfaces
  • Inversely proportional to the # of data types


Orthogonality in Software

  • separation of concerns
  • High cohesion within a module or component
  • low coupling between modules or components
  • little overlap in functionality between modules
  • information hiding / decision hiding

Rotate your perspective 90%

  • When looking at components, think about modules
  • When looking at modules, think about components
  • When looking at data, think about code
  • When looking at code, think about data

Use all your tools

  1. Module structure – layout of your code and libraries
  2. Component structure – interactions between runtime components
  3. Abstraction – Emphasize similar interfaces & data formats

O’Reilly’s Software Architecture – Domain-Driven Design

O’Reilly Learning Course:

Domain Driven Design – Software Architecture

Andrew Harmel-Law

The Heart of Domain-Driven Design

Complexity in Software Tackling complexity is the heart of software. Complexity is in the problem domain, not in our software. We add our own complexity in solving the problem that may or may not be needed. This additional complexity is on us.

Concepts of that are Core to Successful Domain Driven Design

Ubiquitous Language

Domain Experts are developers who can’t code and we are developers who don’t know the domain.

Translation fractures the language between the developer and the business. It’s a mental overhead and things are always lost in translation. If we (the engineering team) and the business have the same understanding and use the same terminology there is no overhead to communication concepts. There is no opportunity to lose meaning in translation. What the point of using two different terms to mean the same thing? Isn’t this idea much like copy and pasting code? They might be the same to start with, but they always diverge over time?

Why are accounts in the banking domain opened and not created?

Respect the domain experts, the domain might seem simple, but once you start talking to the domain experts, it’s not. Domain experts have the solution to one of the hardest problems in Software Development and that’s they names for all the concepts.

Domain experts have a term that succinctly communicates a complex idea?

Are they correcting your word usage?

If so, these are the hints of a complex ideas and the naming to got along with it.

Hands-On Modelers

Developer assumptions get shipped to production and sometimes these assumptions are wrong.

In a typical organization the business domain is documented by a Business Analysis, which records what they think the developers will need. This document is then handed to the Developers. The Developers are reading the Cliff Notes version of the domain problem. This may or may not be the details the developers need to complete their work. Getting the Developers as close to the source of information gives the highest chance of getting the software right.

Engineers working on the model must understand the change they are making and how they are improving the model. If they don’t understand this then they are weakening the model.

Tests verify the facts of your Ubiquitous Language are still true.

Procedural and Data-Centric don’t work for DDD.

Breakthroughs (fine tuning the Ubiquitous Language)

Focus on breakpoints on the Ubiquitous Language. ARea’s of the domain/model that break down and don’t meet the needs of the problem domain.

Tests don’t do this because most of the time you are testing the positive of the assumptions.

Multiple Models

"Bounded Contexts"

A bounded context is an area of the problem domain that had natural or logic borders from other idea’s in a system.

DRY (Don’t Repeat Yourself) doesn’t apply across Bounded Context boundaries. If you do, you are coupling these two Bounded Contexts. The fact that they are two bounded context means they are two different spaces and shouldn’t have overlap. Sharing algothrims is coupling.

If there are truly algorithms that span Bounded Contexts, they should be cross-cutting concerns. For example, working in the financial industry and using financial calculations in multiple Bounded Context is an example of Cross-Cutting concerns.

DRY isn’t about code, it’s not about removing code duplication, but removing duplication of knowledge.

Nested Bounded Contexts aren’t a thing.