Category: Article

4 Practices to Lowering Your Defect Rate

Writing software is a battle between complexity and simplicity. Striking balance between the two is difficult. The trade-off is between long unmaintainable methods and too much abstraction. Tilting too far in either direction impairs code readability and increases the likelihood of defects.

Are defects avoidable? NASA tries, but they also do truckloads of testing. Their software is literally mission critical – a one shot deal. For most organizations, this isn’t the case and large amounts of testing are costly and impractical. While there is no substitute for testing, it’s possible to write defect resistant code, without testing.

In 20 years of coding and architecting applications, I’ve identified four practices to reduce defects. The first two practices limit the introduction of defects and the last two practices expose defects. Each practice is a vast topic on its own in which many books have been written. I’ve distilled each practice into a couple paragraphs and I’ve provided links to additional information when possible.

1. Write Simple Code

Simple should be easy, but it’s not. Writing simple code is hard.

Some will read this and think this means using simple language features, but this isn’t the case — simple code is not dumb code.

To keep it objective, I’m using cyclomatic complexity as a measure. There are other ways to measure complexity and other types of complexity, I hope to explore these topics in later articles.

Microsoft defines cyclomatic complexity as:

Cyclomatic complexity measures the number of linearly-independent
paths through the method, which is determined by the number and
complexity of conditional branches. A low cyclomatic complexity
generally indicates a method that is easy to understand, test, and
maintain.

What is a low cyclomatic complexity? Microsoft recommends keeping cyclomatic complexity below 25.

To be honest, I’ve found Microsoft’s recommendation of cyclomatic complexity of 25 to be too high. For maintainability and complexity, I’ve found the ideal method size is between 1 to 10 lines with a cyclomatic complexity between 1 and 5.

Bill Wagner in Effective C#, Second Edition wrote on method size:

Remember that translating your C# code into machine-executable code is a two-step process. The C# compiler generates IL that gets delivered in assemblies. The JIT compiler generates machine code for each method (or group of methods, when inlining is involved), as needed. Small functions make it much easier for the JIT compiler to amortize that cost. Small functions are also more likely to be candidates for inlining. It’s not just smallness: Simpler control flow matters just as much. Fewer control branches inside functions make it easier for the JIT compiler to enregister variables. It’s not just good practice to write clearer code; it’s how you create more efficient code at runtime.

To put cyclomatic complexity in perspective, the following method has a cyclomatic complexity of 12.

public string ComplexityOf12(int status)
{
    var isTrue = true;
    var myString = "Chuck";

    if (isTrue)
    {
        if (isTrue)
        {
            myString = string.Empty;
            isTrue = false;

            for (var index = 0; index < 10; index++)
            {
                isTrue |= Convert.ToBoolean(new Random().Next());
            }

            if (status == 1 || status == 3)
            {
                switch (status)
                {
                    case 3:
                        return "Bye";
                    case 1:
                        if (status % 1 == 0)
                        {
                            myString = "Super";
                        }
                        break;
                }

                return myString;
            }
        }
    }

    if (!isTrue)
    {
        myString = "New";
    }

    switch (status)
    {
        case 300:
            myString = "3001";
            break;
        case 400:
            myString = "4003";
            break;

    }

    return myString;
}

A generally accepted complexity hypothesis postulates a positive correlation exists between complexity and defects.

The previous line is a bit convoluted. In the simplest terms — keeping code simple reduces your defect rate.

2. Write Testable Code

Studies have shown that writing testable code, without writing the actual tests lowers the incidents of defects. This is so important and profound it needs repeating: Writing testable code, without writing the actual tests, lowers the incidents of defects.

This begs the question, what is testable code?

I define testable code as code that can be tested in isolation. This means all the dependencies can be mocked from a test. An example of a dependency is a database query. In a test, the data is mocked (faked) and an assertion of the expected behavior is made. If the assertion is true, the test passes, if not it fails.

Writing testable code might sound hard, but, in fact, it is easy when following the Inversion of Control (Dependency Injection) and the S.O.L.I.D principles. You’ll be surprised at the ease and will wonder why it took so long to start writing in this way.

3. Code Reviews

One of the most impactful practice a development team can adopt is code reviews.

Code Reviews facilitates knowledge sharing between developers. Speaking from experience, openly discussing code with other developers has had the greatest impact on my code writing skills.

In the book Code Complete, by Steve McConnell, Steve provides numerous case studies on the benefits code reviews:

  • A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a 90 percent decrease in defects after the organization introduced reviews.
    • The Aetna Insurance Company found 82 percent of the errors in a program by using inspections and was able to decrease its development resources by 20 percent.
    • In a group of 11 programs developed by the same group of people, the first 5 were developed without reviews. The remaining 6 were developed with reviews. After all the programs were released to production, the first 5 had an average of 4.5 errors per 100 lines of code. The 6 that had been inspected had an average of only 0.82 errors per 100. Reviews cut the errors by over 80 percent.

If those numbers don’t sway you to adopt code reviews, then you are destined to drift into a black hole while singing Johnny Paycheck’s Take This Job and Shove It.

4. Unit Testing

I’ll admit, when I am up against a deadline testing is the first thing to go. But the benefits of testing can’t be denied as the following studies illustrate.

Microsoft performed a study on the Effectiveness on Unit Testing. They found that coding version 2 (version 1 had no testing) with automated testing immediately reduced defects by 20%, but at a cost of an additional 30%.

Another study looked at Test Driven Development (TDD). They observed an increase in code quality, more than two times, compared to similar projects not using TDD. TDD projects took on average 15% longer to develop. A side-effect of TDD was the tests served as documentation for the libraries and API’s.

Lastly, in a study on Test Coverage and Post-Verification Defects:

… We find that in both projects the increase in test coverage is
associated with decrease in field reported problems when adjusted for
the number of prerelease changes…

An Example

The following code has a cyclomatic complexity of 4.

    public void SendUserHadJoinedEmailToAdministrator(DataAccess.Database.Schema.dbo.Agency agency, User savedUser)
    {
        AgencySettingsRepository agencySettingsRepository = new AgencySettingsRepository();
        var agencySettings = agencySettingsRepository.GetById(agency.Id);

        if (agencySettings != null)
        {
            var newAuthAdmin = agencySettings.NewUserAuthorizationContact;

            if (newAuthAdmin.IsNotNull())
            {
                EmailService emailService = new EmailService();

                emailService.SendTemplate(new[] { newAuthAdmin.Email }, GroverConstants.EmailTemplate.NewUserAdminNotification, s =>
                {
                    s.Add(new EmailToken { Token = "Domain", Value = _settings.Domain });
                    s.Add(new EmailToken
                    {
                        Token = "Subject",
                        Value =
                    string.Format("New User {0} has joined {1} on myGrover.", savedUser.FullName(), agency.Name)
                    });
                    s.Add(new EmailToken { Token = "Name", Value = savedUser.FullName() });

                    return s;
                });
            }
        }
    }

Let’s examine the testability of the above code.

Is this simple code?

Yes, it is, the cyclomatic complexity is below 5.

Are there any dependencies?

Yes. There are 2 services AgencySettingsRepository and EmailService.

Are the services mockable?

No, their creation is hidden within the method.

Is the code testable?

No, this code isn’t testable because we can’t mock AgencySettingsRepository and EmailService.

Example of Refactored Code

How can we make this code testable?

We inject (using constuctor injection) AgencySettingsRepository and EmailService as dependencies. This allows us to mock them from a test and test in isolation.

Below is the refactored version.

Notice how the services are injected into the constructor. This allows us to control which implementation is passed into the SendMail constructor. It’s then easy to pass dummy data and intercept the service method calls.

public class SendEmail
{
    private IAgencySettingsRepository _agencySettingsRepository;
    private IEmailService _emailService;


    public SendEmail(IAgencySettingsRepository agencySettingsRepository, IEmailService emailService)
    {
        _agencySettingsRepository = agencySettingsRepository;
        _emailService = emailService;
    }

    public void SendUserHadJoinedEmailToAdministrator(DataAccess.Database.Schema.dbo.Agency agency, User savedUser)
    {
        var agencySettings = _agencySettingsRepository.GetById(agency.Id);

        if (agencySettings != null)
        {
            var newAuthAdmin = agencySettings.NewUserAuthorizationContact;

            if (newAuthAdmin.IsNotNull())
            {
                _emailService.SendTemplate(new[] { newAuthAdmin.Email },
                GroverConstants.EmailTemplate.NewUserAdminNotification, s =>
                {
                    s.Add(new EmailToken { Token = "Domain", Value = _settings.Domain });
                    s.Add(new EmailToken
                    {
                        Token = "Subject",
                        Value = string.Format("New User {0} has joined {1} on myGrover.", savedUser.FullName(), agency.Name)
                    });
                    s.Add(new EmailToken { Token = "Name", Value = savedUser.FullName() });

                    return s;
                });
            }
        }
    }
}

Testing Exmaple

Below is an example of testing in isolation. We are using the mocking framework FakeItEasy.

    [Test]
    public void TestEmailService()
    {
        //Given

        //Using FakeItEasy mocking framework
        var repository = A<IAgencySettingsRepository>.Fake();
        var service = A<IEmailService>.Fake();

        var agency = new Agency { Name = "Acme Inc." };
        var user = new User { FirstName = "Chuck", LastName = "Conway", Email = "chuck.conway@fakedomain.com" }

        //When

        var sendEmail = new SendEmail(repository, service);
        sendEmail.SendUserHadJoinedEmailToAdministrator(agency, user);


        //Then
        //An exception is thrown when this is not called.
        A.CallTo(() => service.SendTemplate(A<Agency>.Ignore, A<User>.Ignore)).MustHaveHappened();

    }

Closing

Writing defect resistance code is surprisingly easy. Don’t get me wrong, you’ll never write defect-free code (if you figure out how, let me know!), but by following the 4 practices outlined in this article you’ll see a decrease in defects found in your code.

To recap, Writing Simple Code is keeping the cyclomatic complexity around 5 and the method size small. Writing Testable Code is easily achieved when following the Inversion of Control and the S.O.L.I.D Principles. Code Reviews help you and the team understand the domain and the code you’ve written — just having to explain your code will reveal issues. And lastly, Unit Testing can drastically improve your code quality and provide documentation for future developers.

Ignorance is Bliss When Using Frameworks

In software engineering, there is a prevailing idea that an engineer should only use a framework when he or she understands the internal workings. This is a fallacy.

Why is it that we must know the internal workings — do the details matter that much? Some might say ignorance is bliss.

Car Engine

Let’s examine the engine of a car:

How many really know how the engine works?

Can you tell me why it’s called a 4 stroke engine?

What does each stroke do?

What’s the difference between a 4 stroke engine and a 2 stroke engine?

Anyone?

And yet we still drive our cars without any thought on “how” the car is getting us to our destination.

We interface with the car using the steering wheel, the gear shifter, the gas pedal, and the brakes.

Who cares how it works, as long as it gets us to our destination. When the car breaks down we take it to an expert.

The Core Competency of a Business


In business, a company has specialized knowledge that allows it to be competitive. This is referred to as a company’s core competency.

A core competency can be a process or a product.

To stay competitive, a company must tirelessly improve their core competency. Using resources for activities other than supporting the company’s core competency weakens the company’s competitive advantage. Which opens the window of opportunity for competitors to overtake the company’s competitive advantage.

This idea is best illustrated with an example.

Apple

Apple is known for their simplicity and their beautiful products. You’d think this would be easy to replicate, but it’s not, just ask Samsung, HTC, and Microsoft.

Why have these companies failed? Because simple is hard and Apple is expert in simple.

The Core Competency of a Person


Core competency can apply to people too.

What sets you apart from others?

To have developed your core competency, you’ve had to rigorously focus in one area, sometime for years, gaining insights and knowledge setting you apart from others.

As in a business, to maintain your competitive advantage you must continually hone your core competency.

Using Small Pieces

A software engineer is no different from a company or any other professional. We must pick and choose what we learn to stay aligned with our core competency.

Understanding the internals of every framework we use is not practical and is time consuming. I’m expecting the framework’s author to be an expert in the framework’s domain, therefore, I don’t need to know it’s internal workings.

Isn’t this the point of software — to use black boxed bits of functionality to produce a larger more complex work? I believe it is.

In the end, it comes down to focus and time, both of which are limited.

8 Must Have Extensions for Brackets.io

Everyone has a favorite editor. We each have reasons for choosing our editor. I’ve tried them all. And I’ve found that Brackets.io best suits me. Unfortunately, there are gaps in the functionality of Brackets.io. With a robust ecosystem of extensions, I’ve found 8 extensions that complete Brackets.io.

Here is a list of my 8 must have extensions.

Emmet

For anyone working with CSS and HTML Emmet is a must have. I wrote about it earlier this year. It removes all the unnecessary typing from while create HTML and CSS.

Autosave

It’s officially called “Autosave Files on Window Blur”. This extension saves all the changes files once you’ve navigated away from Brackets. It works similarly to how WebStorm saves it’s files.

Beautify

You’d think this wasn’t a big deal, at least that’s what I thought. But it does a great job! Give it a try. You’ll be surprised how useful this plugin is — Beautify

Brackets Git

This is the best git integration I have ever used. And I’ve used Git in WebStorm, Sublime Text and Visual Studio. So that’s saying a lot. It’s functional and aesthetically pleasing, there isn’t much else to ask for. – Brackets Git

Brackets Icons

You’d be surprised how much a few good icons can spruce up an ole editor. – Brackets Icons

Documents Toolbar

In my opinion this is a missing feature of Brackets.io. This completes the editor. – Documents Toolbar

Todo

This summarizes all the TODO comments in the file. It also supports NOTE, FIXME, CHANGES and FUTURE. More can be added if this list is too limiting. – Todo

Quick Search

This extension automatically highlights occurrences of the selected word. Much like Notepad++ and Sublime Text. – Quick Search

Right Click Extended

I found the lack of right-click cut and paste annoying. In Windows, right-click cut and paste is bread and butter of my workflow. Out of the box Brackets is missing the right-click cut and paste functionality. This extension saves the day by adding it. – Right-Click Extended

Setting up Continuous Integration on Ubuntu with Nodejs

I went through blood, sweat and tears to bring this to you. I suffered the scorching heat of Death Valley and summited the peaks of Mount McKinley. I’ve sacrificed much.

Much of content shared in the post is not my original work. Where I can, I link back to the original work.

This article assumes you can get around Linux.

I could not find a comprehensive guide on hosting and managing Nodejs applications on Ubuntu in a production capacity. I’ve pulled together multiple articles on the subject. By the end of this article I hope you’ll be able to setup up your own Ubuntu server and have Nodejs deploying via a continuous integration server.

Environment

I am using TeamCity on Windows which then deploys code from GitHub to Ubuntu hosted on AWS.

Technologies

For this article I used the following technologies:

  • Ubuntu 14.04 on AWS
  • Plink 0.64
  • TeamCity 9.1
  • GitHub
  • Nginx 1.9.3

Setting up Ubuntu

I’m not going into detail here. Amazon Web Services (AWS) makes this pretty easy to do. It doesn’t matter where it’s at or if it’s on your own server.

I encountered a few gotchas. First, make sure port 80 is opened. I made the foolish mistake of trying to connect with port 80 closed. Once I discovered my mistake, I felt like a rhinoceros’s ass.

Installing Nodejs From Source

Nodejs is a server technology using Google’s V8 javascript engine. Since it’s release in 2010, its become widely popular.

The following instructions originally came from a Digital Ocean post).

You always have the option to install Nodejs from the apt-get, but it will be a few versions behind. To get the latest bits, install Nodejs from the source.

At this send of this section we will have downloaded the latest stable version of node (as of this article), we will have build the source and installed Nodejs.

Log into your server. We’ll start by updating the package lists.

sudo apt-get update

I’m also suggesting that you upgrade all the packages. This is not necessary, for Nodejs but it is good practice to keep your server updated.

sudo apt-get upgrade

Your server is all up to date. It’s time download the source.

cd ~

As of the writing 12.7 is the latest stable release of Nodejs. Check out nodejs.org for the latest version.

wget https://nodejs.org/dist/v0.12.7/node-v0.12.7.tar.gz

Extract the archive you’ve downloaded.

tar xvf node-v*

Move into the newly created directory

cd node-v*

Configure and build Nodejs.

./configure

make

Install Nodejs

sudo make install

To remove the downloaded and the extracted files. Of course, this is optional.

cd ~

rm -rf node-v*

Congrats! Nodejs is now installed! And it wasn’t very hard.

Setting up Nginx

Source

Nodejs can act as a web server, but it’s not what I would want to expose to the world. An industrial, harden, feature rich web server is better suited for this. I’ve turned to Nginx for this task.

It’s a mature web server with the features we need. To run more than one instance of Nodejs, we’ll need to port forwarding.

You might be thinking, why do we need more than one instance of Nodejs running at the same-time. That’s a fair question… In my scenario, I have one server and I need to run DEV, QA and PROD on the same machine. Yeah, I know not ideal, but I don’t want to stand up 3 servers for each environment.

To start let’s install Nginx

sudo -s

add-apt-repository ppa:nginx/stable

apt-get update 

apt-get install nginx

Once Nginx is has successfully installed we need to set up on the domains. I’m going to assume you’ll want to have each of your sites on it’s own domain/sub domain. If you don’t and want to use different sub-folders, that’s doable and very easy to do. I am not going to cover that scenario here. There is a ton of documentation on how to do that. There is very little documentation on setting up different domains and port forwarding to the corresponding Nodejs instances. This is what I’ll be covering.

Now that Nginx is installed, create a file for yourdomain.com at /etc/nginx/sites-available/

sudo nano /etc/nginx/sites-available/yourdomain.com

Add the following configuration to your newly created file

# the IP(s) on which your node server is running. I chose port 9001.
upstream app_myapp1 {
    server 127.0.0.1:9001;
    keepalive 8;
}

# the nginx server instance
server {
    listen 80;
    server_name yourdomain.com;
    access_log /var/log/nginx/yourdomain.log;

    # pass the request to the node.js server with the correct headers
    # and much more can be added, see nginx config options
    location / {
        proxy_http_version 1.1;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://app_myapp1;

    }
 }

Make sure you replace “yourdomain.com” with your actual domain. Save and exit your editor.

Create a symbolic link to this file in the sites-enabled directory.

cd /etc/nginx/sites-enabled/ 

ln -s /etc/nginx/sites-available/yourdomain.com yourdomain.com

To test everything is working correctly, create a simple node app and save it to /var/www/yourdomain.com/app.js and run it.

Here is a simple nodejs app if you don’t have one handy.

var http = require('http');

http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello World\n');}).listen(9001, "127.0.0.1");
console.log('Server running at http://127.0.0.1:9001/');

Let’s restart Nginx.

sudo /etc/init.d/nginx restart

Don’t forget to start your Nodejs instance, if you haven’t already.

cd /var/www/yourdomain/ && node app.js

If all is working correctly, when you navigate to yourdomain.com you’ll see “Hello World.”

To add another domain for a different Nodejs instance your need to repeat the steps above. Specifically you’ll need to change the upstream name, the port and the domain in your new Nginx config file. The proxy_pass address must match the upstream name in the nginx config file. Look at the upstream name and the proxy_pass value and you’ll see what I mean.

To recap, we’ve installed NodeJS from source and we just finished installing Nginx. We’ve configured and tested port forwarding with Nginx and Nodejs

Installing PM2

You might be asking “What is PM2?” as I did when I first heard about. PM2 is a process manager for Nodejs applications. Nodejs doesn’t come with much. This is part of it’s appeal. The downside to this, is well, you have to provide the layers in front of it. PM2 is one of those layers.

PM2 manages the life of the Nodejs process. When it’s terminated, PM2 restarts it. When the server reboots PM2 restarts all the Nodejs processes for you. It also has extensive development lifecycle process. We won’t be covering this aspect of PM2. I encourage you to read well written documentation.

Assuming you are logged into the terminal, we’ll start by installing PM2 via NPM. Npm is Nodejs package manager (npm). It was installed when you installed Nodejs.

sudo npm install pm2 -g

That’s it. PM2 is now installed.

Using PM2

PM2 is easy to use.

The hello world for PM2 is simple.

pm2 start hello.js

This adds your application to PM2’s process list. This list is output each time an application is started.

In this example there are two Nodejs applications running. One called api.dev and api.pre.

PM2 automatically assigns the name of the app to the “App name” in the list.

Out of the box, PM2 does not configure itself to startup when the server restarts. The command is different for the different flavors of Linux. I’m running on Ubuntu, so I’ll execute the Ubuntu command.

pm2 start ubuntu

We are not quite done yet. We have to add a path to the PM2 binary. Fortunately, the output of the previous command tells us how to do that.

Output:

[PM2] You have to run this command as root
[PM2] Execute the following command :
[PM2] sudo env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u sammy
Run the command that was generated (similar to the highlighted output above) to set PM2 up to start on boot (use the command from your own output):

 sudo env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u sammy

Examples of other PM2 usages (optional)

Stopping an application by the app name

pm2 stop example

Restarting by the app name

pm2 restart example

List of current applications managed by PM2

pm2 list

Specifying a name when starting a process. If you call, PM2 uses the javascript file as the name. This might not work for you. Here’s how to specify the name.

pm2 start www.js --name api.pre

That should be enough to get you going with PM2. To learn more about PM2’s capabilities, visit the GitHub Repo.

Setting up and Using Plink

You are probably thinking, “What in the name of Betsey’s cow is Plink?” At least that’s thought. I’m still not sure what to think of it. I’ve never seen anything like it.

You ever watched the movie Wall-e? Wall-e pulls out a spork. First he tries to put it with the forks, but it doesn’t fix and then he tries to put it with the spoons, but it doesn’t fit. Well that’s Plink. It’s a cross between Putty (SSH) and the Windows Command Line.

Plink basically allows you to run bash commands via the Windows command line while logged into a Linux (and probably Unix) shell.

Start by downloading Plink. It’s just an executable. I recommend putting it in C:/Program Files (x86)/Plink. We’ll need to reference it later.

If you are running an Ubuntu instance in AWS. You’ll already have a cert setup for Putty (I’m assuming you are using Putty).

If you are not, you’ll need to ensure you have a compatible ssh cert for Ubuntu in AWS.

If you are not using AWS, you can specify the username and password in the command line and won’t to worry about the ssh certs.

Here is an example command line that connects to Ubuntu with Plink.

"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk" 

This might be getting ahead of ourselves, but to run an ssh script on the Ubuntu server we add the complete path to the end of the Plink command.

"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk" /var/www/deploy-dev-ui.sh

And that, dear reader, is Plink.

Understanding NODE_ENV

NODE_ENV is an environment variable made popular by expressjs. Before your start the node instance, set the NODE_ENV to the environment. In the code you can load specific files based on the environment.

Setting NODE_ENV
Linux & Mac: export NODE_ENV=PROD
Windows: set NODE_ENV=PROD

The environment variable is retrieved inside a Nodejs instance by using process.env.NODE_ENV.

example

var environment = process.env.NODE_ENV

or with expressjs

app.get('env')

*Note: app.get(‘env’) defaults to “development”.

Bringing it all together

Nodejs, PM2, Nginx and Plink are installed and hopefully working. We now need to bring all these pieces together into a continuous integration solution.

Clone your GitHub repository in /var/www/yourdomain.com. Although SSH is more secure than HTTPS, I recommend using HTTPS. I know this isn’t ideal, but I couldn’t get Plink working with GitHub on Ubuntu. Without going into too much detail Plink and GitHub SSH cert formats are different and calling GitHub via Plink through SSH didn’t work. If you can figure out the issue let me know!

To make the GitHub pull handsfree, the username and password will need to be a part of the origin url.

Here’s how you set your origin url. Of course you’ll need to substitute your information where appropriate.

git remote set-url origin  https://username:password@github.com/username/yourdomain.git

Clone your repository.

cd /var/www/yourdomain.com
git clone https://username:password@github.com/username/yourdomain.git .

Note, that if this directory is not completely empty, including hidden files Git will not clone the repo to this directory.

To find hidden files in the directory run this command

ls -a

For the glue, we are using a shell script. Here is a copy of my script.

#!/bin/bash

echo "> Current PM2 Apps"
pm2 list

echo "> Stopping running API"
pm2 stop api.dev

echo "> Set Environment variable."
export NODE_ENV=DEV

echo "> Changing directory to dev.momentz.com."
cd /var/www/yourdomain.com

echo "> Listing the contents of the directory."
ls -a

echo "> Remove untracked directories in addition to untracked files."
git clean -f -d

echo "> Pull updates from Github."
git pull

echo "> Install npm updates."
sudo npm install

echo "> Transpile the ECMAScript 2015 code"
gulp babel

echo "> Restart the API"
pm2 start transpiled/www.js --name api.dev

echo "> List folder directories"
ls -a

echo "> All done."

I launch this shell script with TeamCity, but you can launch with anything.

Here is the raw command.

"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk" /var/www/deploy-yourdomain.sh
exit
>&2

That’s it.

In Closing

This process has some rough edges… I hope to polish those edges in time. If you have suggestions please leave them in the comments.

This document is in my GitHub Repository. Technologies change, so if you find an error please update it. I will then update this post.

The Mind State of a Software Engineer

Have patience.

Coding is discovery. Coding is failing. Be ok with this.

*image reference

Don’t blame the framework. It’s more probable it’s your code. Accept this fallibility.

*image reference

Know when to walk away. You mind is a wonderful tool, even at rest it’s working on unsolved problems. Rest, and let your mind do it’s work.

*image reference

Be comfortable not knowing. Software engineering is a vast ocean of knowledge. Someone will always know more than you. The sooner you are OK with this the sooner you will recognize the opportunity to learn something new.

*image reference

Anger and frustration don’t fix code. Take a break, nothing can be accomplished in this state.

*image reference

3 Reasons Why Code Reviews are Important

A great code review will challenge your assumptions and give you constructive feedback. For me, code reviews are an essential part in growing as a software engineer.

Writing code is an intimate process. Software engineers spend years learning the craft of software engineering and when something critical is said of our creation it’s hard not to take it personal. I find myself, at times, getting defensive when I hearing criticisms. I know the reviewer means well, but this isn’t always comforting. If it wasn’t for honest feedback from some exceptional software engineers, I wouldn’t be half the software engineer I am today.

Benefits of Code Reviews

1. Finding Bugs

Sometimes it’s the simple fact of reading the code that you find an error. Sometimes it’s the other developer who spots the error. Regardless, simply walking the code is enough to expose potential issues.

I think of my mistakes as the grindstone to my sword. To quote Michael Jordan:

I’ve missed more than 9000 shots in my career. I’ve lost almost 300 games. 26 times, I’ve been trusted to take the game winning shot and missed. I’ve failed over and over and over again in my life. And that is why I succeed.

2. Knowledge Transfer

Sharing your work with others is humbling. In many ways you are the code. I know that I feel vulnerable when I share my code.

This a great opportunity to learn from and to teach other engineers. In sharing your code you are taking the reviews on a journey, a journey into the code and aspects about you. A lot can be learned about you by how your write code.

At the end of the code review the reviewers should have a good understanding of how the code works, the rationale behind it and will have learned a little bit about you.

3. Improving the Health of the Code

As I mentioned, the more times you read the code the better code becomes. The more reviewers the better the chance one of them will suggest an improvement. Some might think skill level matters, it doesn’t. Less experienced software engineers don’t have the deep technological knowledge as experienced software engineers, but they also don’t have to wade through all the mental technical baggage to see opportunities for improvement.

Code reviews gives us the benefit of evaluating our code. There will always be something to change to make it just a little bit better.

Coding, in this way, is much like writing. For a good piece to come into focus the code must rest and be re-read. The more times you repeat this process the better the code will become.

In Closing

Some companies don’t officially do code reviews, that’s ok. Seek out other engineers. Most software engineer’s will be happy to take 10 to 15 minutes to look over your code.

5 Steps for Coding for the Next Developer

Most of us probably don’t think about the developer who will maintain our code. Until recently, I did not consider him either. I never intentionally wrote obtuse code, but I also never left any breadcrumbs.

Kent Beck on good programmers:

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

Douglas Crockford on good computer programs:

It all comes down to communication and the structures that you use in order to facilitate that communication. Human language and computer languages work very differently in many ways, but ultimately I judge a good computer program by it’s ability to communicate with a human who reads that program. So at that level, they’re not that different.

Discovering purpose and intent is difficult in the most well written code. Any breadcrumbs left by the author, comments, verbose naming and consistency, is immensely helpful to next developers.

I start by looking for patterns. Patterns can be found in many places including variables names, class layout and project structure. Once identified, patterns are insights into the previous developer’s intent and help in comprehending the code.

What is a pattern? A pattern is a repeatable solution to a recurring problem. Consider a door. When a space must allow people to enter and to leave and yet maintain isolation, the door pattern is implemented. Now this seems obvious, but at one point it wasn’t. Someone created the door pattern which included the door handle, the hinges and the placement of these components. Walk into any home and you can identify any door and it’s components. The styles and colors might be different, but the components are the same. Software is the same.

There are known software patterns to common software problems. In 1995, Design Patterns: Elements of Reusable Object-Oriented Software was published describing common software patterns. This book describes common problems encountered in most software application and offered an elegant way to solve these problems. Developers also create their own patterns while solving problems they routinely encounter. While they don’t publish a book, if you look close enough you can identify them.

Sometimes it’s difficult to identify the patterns. This makes grokking the code difficult. When you find yourself in this situation, inspect the code, see how it is used. Start a re-write. Ask yourself, how would you accomplish the same outcome. Often as you travel the thought process of an algorithm, you gain insight into the other developer’s implementation. Many of us have the inclination to re-write what we don’t understand. Resist this urge! The existing implementation is battle-tested and yours is not.

Some code is just vexing, reach out to a peer — a second set of eyes always helps. Walk the code together. You’ll be surprised what the two of you will find.

Here are 5 tips for leaving breadcrumbs for next developers

1. Patterns
Use known patterns, create your own patterns. Stick with a consistent paradigm throughout the code. For example, don’t have 3 approaches to data access.

2. Consistency
This is by far the most important aspect of coding. Nothing is more frustrating than finding inconsistent code. Consistency allows for assumptions. Each time a specific software pattern is encountered, it should be assumed it behaves similarly as other instances of the pattern.

Inconsistent code is a nightmare, imagine reading a book with every word meaning something different, including the same word in different places. You’d have to look up each word and expend large amounts of mental energy discovering the intent. It’s frustrating, tedious and painful. You’ll go crazy! Don’t do this to next developer.

3. Verbose Naming
This is your language. These are the words to your story. Weave them well.

This includes class names, method names, variable names, project names and property names.

Don’t:

if(monkey.HoursSinceLastMeal > 3)
{
    FeedMonkey();
}

Do:

int feedInterval = 3;

if(monkey.HoursSinceLastMeal > feedInterval)
{
    FeedMonkey();
}

The first example has 3 hard coded in the if statement. This code is syntactically correct, but the intent of the number 3 tells you nothing. Looking at the property it’s evaluated against, you can surmise that it’s really 3 hours. In reality we don’t know. We are making an assumption.

In the second example, we set 3 to a variable called ‘feedInterval’. The intent is clearly stated in the variable name. If it’s been 3 hours since the last meal, it’s time to feed the monkey. A side effect of setting the variable is we can now change the feed interval without changing the logic.

This is a contrived example, in a large piece of software this type of code is self documenting and will help the next developer understand the code.

4. Comments
Comments are a double edge sword. Too much commenting increases maintenance costs, not enough leaves developers unsure on how the code works. A general rule of thumb is to comment when the average developer will not understand the code. This happens when the assumptions are not obvious or the code is out of the ordinary.

5. Code Simple
In my professional opinion writing complex code is the biggest folly among developers.

Steve Jobs on simplicity:

Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end because once you get there, you can move mountains.

Complexity comes in many forms, some of which include: future proofing, overly complex implementations, too much abstraction, large classes and large methods.

For more on writing clean simple code, see Uncle Bob’s book Clean Code and Max Kanat-Alexander’s Code Simplicity

Closing

Reading code is hard. With a few simple steps you can ensure the next developer will grok your code.

A Simple Guide to Finding Your Next Job

It’s time to look for the next job, eh? I feel for you. Finding a job sucks. It’s one of those things that everyone must do at some point. I equate it to looking for love. Every aspect of “you” is on display. When someone passes on you, it’s hard not to take it personally. Chin up my friend; we’ll get through this.

Who am I? Good question. I’m a software consultant, I’ve had three jobs, on average, each year for the last four years. I’ve been on both sides of the table in hundreds of interviews. I’ve learned what works and what does not.

1. Your Resume

Before you start applying for jobs a good resume is a must. A resume speaks to who you are and what value you offer an employer. Your resume is your ticket to an interview. In fact, the sole purpose of a resume is to get an interview.

This is so important; I need to repeat it: “The sole purpose of a resume is to get an interview.”

Resume Tips
There is always an urge to fib a little, who will know, right? Don’t lie on your resume. I’ve interviewed candidates who padded their resume with skills without knowing how to explain or perform these skills. I’ll add a caveat and say that your resume needs to be a simple read. If you held a position called System Analyst III, which had the responsibilities of a Senior Software Engineer then change the title to Senior Software Engineer otherwise, a prospective employer would decipher your job titles instead of checking your qualifications.

Compare your resume to the job description. Make sure your resume is a good fit for the position. Sometimes this might require moving some items around on your resume to highlight the desired skills for the position.

A little statistic: On average, a hiring manager will look at your resume somewhere between 3 and 15 seconds. Make that time count!

Keep Your Resume Updated
Even when not looking for a job, it’s good to update your resume. When the time comes to hand out your resume, you won’t be left trying to remember your accomplishments.

Your Resume’s Presentation

Chefs know presentation. A meal that looks unappetizing, regardless of taste, goes uneaten.

Make you resume visually appealing. It needs to be clean, consistent and well structured. It might behoove you to hire a designer. You want your resume to standout from the rest. Be mindful of the audience. A graphic designer’s resume looks entirely different than a database engineer’s resume.

2. Getting an Interview

Your resume is polished. It’s time to start applying for positions.

I start with the following sites:

Dice
This is my go-to site for technology-related positions. Recruiters or companies will see your resume. The downside to a large job site like Dice is recruiters will inquire about positions outside your geographic area or they will inquire about positions not matching your skill-set.

Careers Stackoverflow
This is the cream of the crop. Companies posting positions on Careers are looking for the best and the brightest. I’d pay close attention to these companies.

Craigslist
This is a hit or miss. Some of the postings are good; others are not. It’s still worth a look. You might find a great opportunity.

Indeed.com
Indeed searches many companies and job sites. They discover jobs not usually found on other popular job sites.

Linkedin
If you are not a member, sign up now! It’s the Facebook of the professional world. There is a search that will return positions not found on the other sites.

Friends and Family
Put the word out with your friends. You’ll have more credibility if you are recommended by someone the hiring manager already knows. I put a signature at the bottom of my emails stating what I do and how to contact me.

Cold Calling Companies
I’ve never done this, but if you have time on your hands why not?

Recruiters
Companies tend to hire agencies to find upper-level management or highly technical candidates. It’s good to check with a recruiter if possible build a relationship with a couple of them.

Finding the right job takes time. It’s not uncommon for the search to take a couple of months or even a year. Don’t get discouraged. Take a temporary position if you must, and do your best to keep your skills sharp while you keep looking for a permanent position in your field.

Recruiters are sales people. Their job is to convince you to take the position. Recruiters will try to talk your rate down. It’s their job! Be ready for it.

Know Your Value
Know your value, go to salary.com or glassdoor.com to find the rate of compensation in your area. I have friends who make over 200k a year. When the market says, they should be making 100k a year. How do they do it? They know their value and stick to their guns when pressured. On the downside, occasionally a position is lost. Someone who needs a position might not want to risk losing an opportunity. Taking a lower rate might be the best thing for your situation. This is something only you can determine.

Know What Positions You’ve Applied To
Track where you have applied, it can benefit you in two ways. First, applying to a position twice gives the appearance of an unorganized person and second, you can follow up with companies where you’ve applied.

3. The Interview

Congratulations! You have an interview!

The interview is the most important part of the job hiring process. It’s also the most nerve-racking. To have a successful interview, confidently articulating your ideas is a must. This is, of course, easier said than done. Being prepared will help immensely. This starts with knowing your resume inside and out. Expect questions on every aspect of your resume. If you are nervous in interviews, practice interviewing in front of a mirror or with a friend.

When responding to questions, make your answers concise. If you don’t know the answer, don’t waste time talking about something you don’t know. An interviewer can spot this a mile away. Instead, simply say “I don’t know”. Sometimes you may have an educated guess, share this with the interviewer.

Interviewing is a two-way street. The company is evaluating you. Use this time to do the same. Always put together a list of questions to ask during the interview. The list should include questions on their development process, the position’s responsibilities, the project, the work environment and the company policies (i.e. hours, telecommuting, etc.). The aim is to walk away from the interview with an understanding of the company, the position’s expectations and the goals of the project.

Unfortunately, sometimes your best just is not enough, there are candidates out there that are better qualified (i.e. more education, more experience…, etc. ). There is nothing wrong with this. It happens. Don’t let it get you down.

Following Up
Most companies will give feedback within a couple of days. If you don’t hear from them within a week, reach out to your contact and inquire about the status of the position.

In Closing
Searching for a job is challenging. It may take many interviews to land the right position.
When you do not see the results you expect, don’t be afraid to change up your strategy. Most importantly, stay positive. This can be hard at times, especially when you receive rejection letter after rejection letter. When you do finally land the position, it will be worth it.

When are you a Senior Developer?

When I hear “Senior Developer” I think of someone who has mastered programming. I think of a person who can design, code and test a system. They can talk to system architecture or component design. They understand and use design patterns. This person can anticipate the performance bottlenecks, but knows not to pre-optimize. This person will leverage asynchronous programming, queuing, caching, logging, security and persistence when appropriate. When asked they can give a detail explanation of their choice and the pros and cons. In most cases they have mastered object oriented programming and design, this not an absolute other languages such as javascript, F#, scheme are powerful and are not object oriented at heart. They are adept in risk management and most important of all they can communicate the before mentioned to their peers.

What is mastery? There is a generally accepted idea, that to master ANY one skill it takes 10,000 hours of repetition for the human body and mind to grasp and internalize a skill. This is written to at length in Malcolm GladWell’s book Outliers.

Some examples of in Malcolm GladWell’s Outliers are:

Mozart his first concerto at the young age of 21. Which at first seems young, but he has been writing music since he was 11 years old.

The Beatles were initially shunned. They were told they did not have the mustard and should consider a different line of work. They spend 3 years in Germany playing about 1200 times at different venues, each time being 5 to 8 hours in length. They re-emerged as The Beatles we know and love today.

And lastly, Bill Gates at age 20 dropped out of Harvard to found Microsoft. To some this might seem foolish, but considered at 20 he had spent nearly half of his young life programming. In 1975, only maybe 50 people in the world had the experience he did. His experience gave him the foresight to see the future in Microsoft.

Peter Norvig also discusses the 10,000 hours rule in his essay “Teach Yourself Programming in Ten Years”.

In the book Mastery by George Leonard, great detail is given on how to master a skill. One must practice the skill over and over and over again. The more the repetition, the more you become aware of the differences in each repetition. Only with this insight can you become better.

The software industry’s titles (Junior, Mid-Level and Seniors) are misleading and inconsistent from organization to organization. I’ve worked with companies, who defined a Senior Developer as someone whom had 5 years or more of experience. There is not mention to the quality of the experience, just that they have sat in from of a computer for 5 years. In working with these folks many of them had not yet grasp object oriented programming, yet they were considered Senior Developers.

There must be a better more objective way to measure the skill set of a software engineer. John Haugeland has come up with a computer programmer’s skills matrix. It’s gives a common, objective way to measure a programmer’s skill level, which otherwise is left mostly to gut feeling.

When looking at software engineers I see 4 tiers of skills: Luminary, Senior, Mid-Level and Junior.

Liminary (10+ years) is one who has mastered a skill and has set about improving their respective discipline. Some examples include: Ted Neward, Uncle Bob Martin, Donald Knuth, Oren Eini, Peter Norvig, Linus Torvalds. This is change depending on your skill-set.

Senior (7 to 10+ years, Level 3) is one who has spent the last 10,000 hours programing in a specific genre. There is a strong understanding of design patterns, They leverage asynchronous programming, queuing, caching, logging, security and persistence when appropriate.

It’s very possible that a Senior will never reach Liminary. Liminary’s are often found speaking and writing. They are actively trying to impact their discipline.

Mid-Level (4 to 6 years, Level 2) is one who understands day to day programming. They work independently and create robust solutions. However they have yet to experience creating or maintaining large or complex systems. In general Mid-Level developers are great with component level development.

Junior (1 to 3 years, Level 1) is one who understands the basics of programming. They either have a degree in software engineering or computer science or they are self taught. Their code is continually reviewed. Guidance is given in regards to algorithms, maintainability and structure.

Setting up Single Sign On with Windows 2012 and ASP.Net MVC 4

Requirements

  • Windows 2012
  • AD FS 2.0 Feature installed
  • ASP.Net MVC 4.0
  • Valid SSL certs
  • Visual Studio 2012
    This document covers setting up an ASP.NET MVC 4.0 application using Visual Studio 2012, Windows 2012 and AD FS 2.0 to enable Web Single Sign On.

It’s important to have valid SSL certificates. Self signed certificates will not work. SSL certificates are used to encrypt the tokens and will not work with self-signed certificates. This is very important. If you do not have valid certificates this will not work. Don’t waste your time without valid certificates.

Setting up AD FS 2.0 on Windows 2012

Assuming Windows 2012 is installed. With a valid SSL certificate install the SSL certificate in IIS. This is done by opening IIS (7+) Management Console, selecting the root web server node and opening the Server Certificates found in the Feature view. Do not install the AD FS feature before installing the certificate. AD FS extracts the host name from the SSL certificate and will use localhost if a certificate is not found.

If AD FS is installed before installing the SSL certificates

If this does happen, you’ll need to uninstall the the AD FS role and manually delete the IIS applications (removing them from the IIS management console is not enough they must be removed from the IIS metabase via the command line).

C:WindowsSystem32inetsrvappcmd.exe delete app "Default Web Site/adfs"
C:WindowsSystem32inetsrvappcmd.exe delete app "Default Web Site/adfs/ls"

Once a valid certificate is installed the AD FS role can be installed.

When reinstalling ADFS, the relying party needs to rebind to AD FS’s FederationMetadata otherwise you’ll encounter an ASP.Net error stating there was a token error.

Federation Services URL

At times this url uses localhost as the host. I am not certain of the cause, but unless localhost is the domain used in the ssl cert, it will not work. The following link describes how to change it.
http://technet.microsoft.com/en-us/library/dd353709%28v=ws.10%29.aspx

At this point AD FS is setup on the server. However, it does not have any trusts established, with the exception of Active Directory (configured by default). The next step is to create a trust with a Relying party.

A trust is a relationship setup between the relying party(ASP.Net MVC) and the issuer (AD FS). The trust is setup on both the relying party and the issuer.

https://myhostname.com/federationmetadata/2007-06/federationmetadata.xml

Creating a Relying Party

Assuming you have installed Visual Studio 2012, download the Windows Identity Foundation Identity and Access Tool Extension tools. Once installed create a MVC 4 project. The option “Identity and Access” is added to the project right-click menu. The Windows Identity Foundation SDK might also be a requirement.

This brings up the Identity and Access option screen.

Enter the path to the STS metadata document

The STS metadata document is generated by the AD FS server. The FederationMetadata.xml defines the Issuer (sometimes referred to as the Identity Server) and allows MVC 4(Relying Party) to create a trust between itself and the Issuer.

Enter the Realm for your Application

The realm is the MVC application. Unless the MVC application and the AD FS are on the same server the localhost host will not work. The realm is the url to your site.

Setting up a Relying Party Trust in AD FS

The Relying Party is created, now it’s time to set up the Relying Party Trust in AD FS. Back on the server open up AD FS MMC screen and click on “Add Relying Party Trust…”

The next screen asks for the FederationMetadata.xml from the relying party. In the previous step when the FederationMetadata.xml was added into the MVC 4 application a FederationMetadata.xml was created for the MVC 4 application. Now we must import the Relying Party FederationMetadata.xml into the AD FS server to complete the trust.

The FederationMetadata.xml can either be imported via a url or added via a local file. The default FederationMetadata.xml path is https://myhostname.com/FederationMetadata/2007-06/FederationMetadata.xml (same as it was for the AD FS server). If it successfully retrieves the FederationMetadata.xml you can click next until the end.

The next step is to add an endpoint. The endpoint is where a user is redirected to upon successful authentication. Start by clicking the Relying Party Trusts folder, you’ll see the Relying Party Trust that was just created. Right-click on it and select “Properties” a tabbed interface will appear. From the monitoring tab Uncheck “Automatically update relying party” this feature does not work out of the box.

Note: If the relying party is configured correctly this step is not necessary.

Click on the endpoint tab and click add. The “Add an Endpoint” dialog will appear. Select WS-Federation for the Endpoint Type. This will automatically set the Binding dropdown to “POST”. In the URL field enter the url in which an authenticated user is sent.

That’s it! The AD FS single sign on should prompt you for credentials. Once entered it will redirect you back to your site’s landing page.

Accessing AD FS with a non-IE browser

Out of the box you’ll encounter a dialog asking for username and password. For whatever reason it would not accept my domain credentials.

In a nutshell a property ExtendedProtectionTokenCheck is set to required when running windows 7. Only IE supports this feature.

More information on the issue
http://stackoverflow.com/questions/6309210/ntlm-authentication-to-ad-fs-for-non-ie-browser-without-extended-protection-sw

How to disable it
http://social.technet.microsoft.com/wiki/contents/articles/1426.ad-fs-2-0-continuously-prompted-for-credentials-while-using-fiddler-web-debugger.aspx

Troubleshooting Tips

1. Recompile the website and trying adding the trust for both ADFS and the web site.

2. Re-install AD FS, that had some issues also, but sometimes it getting to get a fresh start.

3. Installing ADFS Multiple times and you encounter a 503 on the FederationServerServices.asmx

Try removing the ACLs

netsh http delete urlacl url=http://+:80/adfs/services/
netsh http delete urlacl url=https://+:443/adfs/services/
netsh http delete urlacl url=https://+:443/FederationMetadata/2007-06/
netsh http delete urlacl url=https://+:443/adfs/fs/federationserverservice.asmx/

Continuing with the tips

4. When reinstalling ADFS after IIS was removed, the previous ADFS web directory under C:inetpubadfs* needs to be deleted.

5. The configuration service URL 'net.tcp://localhost:1500/policy' may be incorrect or the AD FS 2.0 Windows Service is not running.

setspn -l myservername

before

The SPN is not set or is incorrect. The following thread discusses it in more detail.

http://social.technet.microsoft.com/Forums/en-US/winserverDS/thread/cd9bc625-49f3-499b-9bf3-4ef32fbf64ec/

hint: casing does matter.

After

6. The X.509 certificate CN=ADFS Signing - mydomain.com is not in the trusted people store. The X.509 certificate CN=ADFS Signing - mydomain.com chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.

This is not recommended for going into production, but it will get you past the issue:

http://social.technet.microsoft.com/wiki/contents/articles/windows-identity-foundation-wif-how-to-change-certificate-chain-validation-settings-for-web-applications.aspx

In Conclusion

My overall experience with AD FS 2.0 and Windows 2012 has been painful. Working with AD FS 2.0 reminds me of working with Sharepoint 2010 in the early days.

Some of the configuration is done differently on the Windows 2012 server versus in Visual Studio, even though you are doing the same exact things (i.e. setting up a trust). Once the server is setup and the trusts are established and working correct it’s a thing of beauty, until you get there, good luck!