Building Inspiring Software
If you must run code in a constructor. I’d look for a different way, but if you must,
public class MyClass
{
public MyClass()
{
Task.Run(async () => {
var result = await AsyncActivity();
});
}
}
This is shamelessly stolen from The Humble Developer.
List All Containers
docker ps -a
Stop All Running Containers
docker stop $(docker ps -aq)
Remove All Containers
docker rm $(docker ps -aq)
Remove All Images
docker rmi $(docker images -q)
This was one of the more frustrating issues with Angular 2/4/+. It’s not an issue with Angular 2/4/+ per se, but with how webpack bundles the supporting HTML files.
This issue happens when you run webpack with the production flag (webpack -p). The compilation completes, but running the site generates a runtime error: Template parse errors
[generic]
Uncaught Error: Template parse errors:
Unexpected closing tag “a”. It may happen when the tag has already been closed by another tag. For more info see https://www.w3.org/TR/html5/syntax.html#closing-elements-that-have-implied-end-tags (“ Home
[/generic]
The suspected HTML:
[html]
[/html]
I couldn’t find anything wrong with this HTML. I ran it through multiple online HTML validators, and the HTML always came back as valid HTML. After a few weeks of beating my head against the wall and getting nowhere, I figured there must be a way around this issue.
What I discovered, via a GitHub thread (sorry I lost the link to it) was it’s the HTML loader trying to minimize the HTML that’s causing the problem. If the HTML minimization is turned off, this issue goes away. To be fair, most folks with this error had invalid HTML. I encourage you to check your HTML before working around this issue, you may be just pushing the issue down the road.
Turning off HTML minimization is easy as updating your HTML-loader in the webpack.config.js.
Before:
[js]
{
test: /.html$/,
loader: ‘html-loader’
},
[/js]
After:
[js]
{
test: /.html$/,
use: [{
loader: ‘html-loader’,
options: {
minimize: false,
removeComments: false,
collapseWhitespace: false,
removeAttributeQuotes: false,
}
}],
},
[/js]
One of the selling points of the Single Page Application (SPA) was offloading work traditionally performed on the server onto the client. I feel the SPA has delivered on this promise.
However, it’s not all roses. It’s easy to get overzealous and push too much work to the client. We often forget that we can’t control the client’s environment — the client can be anything from a ten-year-old machine to the latest-and-greatest smartphone with access to varying internet speeds. The only thing we can count on is the user viewing our site in a browser.
In my experience, intensive processing and data needing consistency between the server and the client, such as date conversions or data requiring precise calculations, such as money, are prime candidates for server-side rendering.
A common task performed on the client is paging. With small datasets this works great; however, small datasets never stay small. As data grows, the application slows and eventually becomes unresponsive. Unfortunately, you don’t know it’s happening because it’s on the client and even worse it doesn’t occur on all clients making it hard to troubleshoot.
Moving paging from the client to the server will alleviate paging related performance issues on the client. You might be thinking, “but now I am making a ton of API calls. Making so many API calls doesn’t seem optimal.” True, it does seem that way, but you’ll be surprised how fast your data returns from the server. And the best part? You control the server and can increase capacity as needed.
At the end of the day, you want to provide the user a rich responsive experience and sometimes that’s letting the server do the heavy lifting.
Summarizing, when possible we want to offload work to the client, but by doing so, we can quickly overburden the client. Keeping arduous tasks such as paging and intensive processing on the server can protect us from overwhelming the client.
For nearly 50 years, the switch statement (also known as the case statement) has been an integral part of programming. In recent years, however, some are claiming that the switch statement has outlived its usefulness. Others go even further by labeling the switch statement as a code-smell.
In 1952, Stephen Kleene conceived the switch statement in his paper, Introduction to Metamathematics. The first notable implementation was in ALGOL 58 in 1958. Later, the switch statement was included in the indelible C programming language, which, as we know, has influenced most modern programming languages.
Fast forward to the present day and virtually every language has a switch statement. However, a few languages have omitted the switch statement. The most notable being Smalltalk.
This piqued my curiosity, why was the switch statement excluded from Smalltalk?
Andy Bower, one of the creators/proponents behind Dolphin Smalltalk shared his thoughts on why Smalltalk excluded the switch statement:
When I first came to Smalltalk from C++, I couldn’t understand how a supposedly fully fledged language didn’t support a switch/case construct. After all when I first moved up to “structured programming” from BASIC I thought that switch was one of the best things since sliced bread. However, because Smalltalk didn’t support a switch I had to look for and understand, how to overcome this deficiency. The correct answer is, of course, to use polymorphism and to make the objects themselves dispatch to the correct piece of code. Then I realized that it wasn’t a “deficiency” at all, but Smalltalk was forcing me into much finer grained OOP design than I had got(ten) used to in C++. If there had been a switch statement available it would have taken me a lot longer to learn this or, worse, I might still be programming C++/Java pseudo-object style in Smalltalk.
I would contend that in normal OOP there is no real need for a switch statement. Sometimes, when interfacing to a non-OOP world (like receiving and dispatching WM_XXXX Windows messages that are not objects but just integers), then a switch statement would be useful. In these situations, there are alternatives (like dispatching from a Dictionary) and the number of times they crop up doesn’t warrant the inclusion of additional syntax.
Was Andy right? Are we better off without the switch statement? Would other languages also benefit from excluding the switch statement?
To shed some light on this question, I’ve put together a comparison between a switch statement, a dictionary, and polymorphism. Let’s call it a smackdown. May the best implementation win!
Each implementation has a method taking one parameter, an integer, and returns a string. We’ll use cyclomatic complexity and maintainability index to examine each implementation. We’ll then take a holistic view of all three implementations.
The code.
| Maintainability Index | 72 |
| Cyclomatic Complexity | 6 |
public class SwitchWithFourCases
{
public string SwitchStatment(int color)
{
var colorString = "Red";
switch (color)
{
case 1:
colorString = "Green";
break;
case 2:
colorString = "Blue";
break;
case 3:
colorString = "Violet";
break;
case 4:
colorString = "Orange";
break;
}
return colorString;
}
}
| Maintainability Index | 73 |
| Cyclomatic Complexity | 3 |
public class DictionaryWithFourItems
{
public string Dictionary(int color)
{
var colorString = "Red";
var colors = new Dictionary<int, string> {{1, "Green"}, {2, "Blue"}, {3, "Violet"}, {4, "Orange"}};
var containsKey = colors.ContainsKey(color);
if (containsKey)
{
colorString = colors[color];
}
return colorString;
}
}
| Total Maintainability Index | 94 |
| Total Cyclomatic Complexity | 15 |
| Maintainability Index | 100 |
| Cyclomatic Complexity | 1 |
public interface IColor
{
string ColorName { get; }
}
| Maintainability Index | 76 |
| Cyclomatic Complexity | 4 |
public class ColorFactory
{
public string GetColor(int color)
{
IColor defaultColor = new RedColor();
var colors = GetColors();
var containsKey = colors.ContainsKey(color);
if (containsKey)
{
var c = colors[color];
return c.ColorName;
}
return defaultColor.ColorName;
}
private static IDictionary<int, IColor> GetColors()
{
return new Dictionary<int, IColor>
{
{1, new GreenColor()},
{2, new BlueColor()},
{3, new VioletColor()},
{4, new OrangeColor()},
{5, new MagentaColor()}
};
}
}
| Maintainability Index | 97 |
| Cyclomatic Complexity | 2 |
public class BlueColor : IColor
{
public string ColorName => "Blue";
}
public class RedColor : IColor
{
public string ColorName => "Red";
}
public class GreenColor : IColor
{
public string ColorName => "Green";
}
public class MagentaColor : IColor
{
public string ColorName => "Magenta";
}
public class VioletColor : IColor
{
public string ColorName => "Violet";
}
Before I dive into the results, let’s define Cyclomatic Complexity and Maintainability Index:
| Cyclomatic Complexity | Maintainability Index | |
|---|---|---|
| Switch Statement | 6 | 72 |
| Dictionary | 3 | 73 |
| Polymorphism | 15 | 94 |
We will examine cyclomatic complexity first.
The results for cyclomatic complexity are straightforward. The dictionary implementation is the simplest. Does this mean it’s the best solution? No, as we’ll see when we evaluate the maintainability index.
Most would think as I did, the implementation with the lowest cyclomatic complexity is the most maintainable — how could it be any other way?
In our scenario, the implementation with the lowest cyclomatic complexity isn’t the most maintainable. In fact in our scenario, it’s the opposite. The most complex implementation is the most maintainable! Mind blown!
If you recall, the higher the maintainability index score, the better. Cutting to the chase, polymorphism has the best maintainability index score — but it also has the highest cyclomatic complexity. What gives? That doesn’t seem right.
Why is the most complex implementation the most maintainable? To answer this, we must understand the maintainability index.
The maintainability index consists of 4 metrics: cyclomatic complexity, lines of code, the number of comments and the Halstead volume. The first three metrics are relatively well known, but the last one, the Halstead Volume, is relatively unknown. Like, cyclomatic complexity, the Halstead Volume attempts objectively measure code complexity.
In simple terms, Halstead Volume measures the number of moving parts (variables, system calls, arithmetic, coding constructs, etc.) in code. The higher the number of moving parts the more complexity. The lower the number of moving parts, the lower the complexity. This explains why the polymorphic implementation scores high on the maintainability index; the classes have little to no moving parts. Another way to look at the Halstead Volume is it measures “moving parts” density.
What is software, if it’s not to change? To reflect the real world, we are introducing change. I’ve added a new color to each implementation.
Below are the revised results.
| Cycolmatic Complexity | Maintainability Index | |
|---|---|---|
| Switch Statement | 7 | 70 |
| Dictionary | 3 | 73 |
| Polymorphism | 17 | 95 |
The switch statement and the polymorphic approaches both increased in cyclomatic complexity by one unit, but interestingly, the dictionary didn’t increase. At first I was puzzled by this, but then I realized the dictionary considers the colors as data and the other two implementations treat the colors as code.I’ll get down to the brass tacks.
Turing our attention to the maintainability index, only one, the switch statement, decreased in maintainability. Polymorphism’s maintainability score improved and yet the complexity also increases (we’d prefer it to decrease). As I mentioned above, this is counter-intuitive.
Our comparison shows that dictionaries can, from a complexity standpoint, scale infinitely. The polymorphic approach is by far the most maintainable and seems to increase in maintainability as more scenarios are added. The switch statement increases in complexity and decreases in maintainability when the new scenario was added. Even before we added the new scenario, it had the worst cyclomatic complexity and maintainability index measures.
Jem Finch from Google shared his thoughts on the switch statements shortcomings:
1. Polymorphic method implementations are lexically isolated from one another. Variables can be added, removed, modified, and so on without any risk of impacting unrelated code in another branch of the switch statement.
2. Polymorphic method implementations are guaranteed to return to the correct place, assuming they terminate. Switch statements in a fall through language like C/C++/Java require an error-prone “break” statement to ensure that they return to the statement after the switch rather than the next case block.
3. The existence of a polymorphic method implementation can be enforced by the compiler, which will refuse to compile the program if a polymorphic method implementation is missing. Switch statements provide no such exhaustiveness checking.
4. Polymorphic method dispatching is extensible without access to (or recompiling of) other source code. Adding another case to a switch statement requires access to the original dispatching code, not only in one place but in every place the relevant enum is being switched on.
5. … you can test polymorphic methods independent of the switching apparatus. Most functions that switch like the example the author gave will contain other code that cannot then be separately tested; virtual method calls, on the other hand, can.6. Polymorphic method calls guarantee constant time dispatch. No sufficiently smart compiler is necessary to convert what is naturally a linear time construct (the switch statement with fall through) into a constant time construct.
Unfortunately, or fortunately, depending on your camp, most languages have a switch statement, and they aren’t going anywhere anytime soon. With this in mind, it’s good to know what’s happening under the hood when compiling switch statements.
There are three switch statement optimizations that can occur:
In an object oriented world the switch statement, conceived in 1952, is a mainstay of the software engineer. A notable exception is Smalltalk where the designers opted to exclude the switch statement.
When compared to alternative equivalent implementations, the dictionary, and polymorphism, the switch statement did not fare as well.
The switch statement is here to stay, but as our comparison has shown there are better alternatives to the switch statement.
The implementations are available on Github.
Writing software is a battle between complexity and simplicity. Striking balance between the two is difficult. The trade-off is between long unmaintainable methods and too much abstraction. Tilting too far in either direction impairs code readability and increases the likelihood of defects.
Are defects avoidable? NASA tries, but they also do truckloads of testing. Their software is literally mission critical – a one shot deal. For most organizations, this isn’t the case and large amounts of testing are costly and impractical. While there is no substitute for testing, it’s possible to write defect resistant code, without testing.
In 20 years of coding and architecting applications, I’ve identified four practices to reduce defects. The first two practices limit the introduction of defects and the last two practices expose defects. Each practice is a vast topic on its own in which many books have been written. I’ve distilled each practice into a couple paragraphs and I’ve provided links to additional information when possible.
Simple should be easy, but it’s not. Writing simple code is hard.
Some will read this and think this means using simple language features, but this isn’t the case — simple code is not dumb code.
To keep it objective, I’m using cyclomatic complexity as a measure. There are other ways to measure complexity and other types of complexity, I hope to explore these topics in later articles.
Microsoft defines cyclomatic complexity as:
Cyclomatic complexity measures the number of linearly-independent
paths through the method, which is determined by the number and
complexity of conditional branches. A low cyclomatic complexity
generally indicates a method that is easy to understand, test, and
maintain.
What is a low cyclomatic complexity? Microsoft recommends keeping cyclomatic complexity below 25.
To be honest, I’ve found Microsoft’s recommendation of cyclomatic complexity of 25 to be too high. For maintainability and complexity, I’ve found the ideal method size is between 1 to 10 lines with a cyclomatic complexity between 1 and 5.
Bill Wagner in Effective C#, Second Edition wrote on method size:
Remember that translating your C# code into machine-executable code is a two-step process. The C# compiler generates IL that gets delivered in assemblies. The JIT compiler generates machine code for each method (or group of methods, when inlining is involved), as needed. Small functions make it much easier for the JIT compiler to amortize that cost. Small functions are also more likely to be candidates for inlining. It’s not just smallness: Simpler control flow matters just as much. Fewer control branches inside functions make it easier for the JIT compiler to enregister variables. It’s not just good practice to write clearer code; it’s how you create more efficient code at runtime.
To put cyclomatic complexity in perspective, the following method has a cyclomatic complexity of 12.
public string ComplexityOf12(int status)
{
var isTrue = true;
var myString = "Chuck";
if (isTrue)
{
if (isTrue)
{
myString = string.Empty;
isTrue = false;
for (var index = 0; index < 10; index++)
{
isTrue |= Convert.ToBoolean(new Random().Next());
}
if (status == 1 || status == 3)
{
switch (status)
{
case 3:
return "Bye";
case 1:
if (status % 1 == 0)
{
myString = "Super";
}
break;
}
return myString;
}
}
}
if (!isTrue)
{
myString = "New";
}
switch (status)
{
case 300:
myString = "3001";
break;
case 400:
myString = "4003";
break;
}
return myString;
}
A generally accepted complexity hypothesis postulates a positive correlation exists between complexity and defects.
The previous line is a bit convoluted. In the simplest terms — keeping code simple reduces your defect rate.
Studies have shown that writing testable code, without writing the actual tests lowers the incidents of defects. This is so important and profound it needs repeating: Writing testable code, without writing the actual tests, lowers the incidents of defects.
This begs the question, what is testable code?
I define testable code as code that can be tested in isolation. This means all the dependencies can be mocked from a test. An example of a dependency is a database query. In a test, the data is mocked (faked) and an assertion of the expected behavior is made. If the assertion is true, the test passes, if not it fails.
Writing testable code might sound hard, but, in fact, it is easy when following the Inversion of Control (Dependency Injection) and the S.O.L.I.D principles. You’ll be surprised at the ease and will wonder why it took so long to start writing in this way.
One of the most impactful practice a development team can adopt is code reviews.
Code Reviews facilitates knowledge sharing between developers. Speaking from experience, openly discussing code with other developers has had the greatest impact on my code writing skills.
In the book Code Complete, by Steve McConnell, Steve provides numerous case studies on the benefits code reviews:
If those numbers don’t sway you to adopt code reviews, then you are destined to drift into a black hole while singing Johnny Paycheck’s Take This Job and Shove It.
I’ll admit, when I am up against a deadline testing is the first thing to go. But the benefits of testing can’t be denied as the following studies illustrate.
Microsoft performed a study on the Effectiveness on Unit Testing. They found that coding version 2 (version 1 had no testing) with automated testing immediately reduced defects by 20%, but at a cost of an additional 30%.
Another study looked at Test Driven Development (TDD). They observed an increase in code quality, more than two times, compared to similar projects not using TDD. TDD projects took on average 15% longer to develop. A side-effect of TDD was the tests served as documentation for the libraries and API’s.
Lastly, in a study on Test Coverage and Post-Verification Defects:
… We find that in both projects the increase in test coverage is
associated with decrease in field reported problems when adjusted for
the number of prerelease changes…
The following code has a cyclomatic complexity of 4.
public void SendUserHadJoinedEmailToAdministrator(DataAccess.Database.Schema.dbo.Agency agency, User savedUser)
{
AgencySettingsRepository agencySettingsRepository = new AgencySettingsRepository();
var agencySettings = agencySettingsRepository.GetById(agency.Id);
if (agencySettings != null)
{
var newAuthAdmin = agencySettings.NewUserAuthorizationContact;
if (newAuthAdmin.IsNotNull())
{
EmailService emailService = new EmailService();
emailService.SendTemplate(new[] { newAuthAdmin.Email }, GroverConstants.EmailTemplate.NewUserAdminNotification, s =>
{
s.Add(new EmailToken { Token = "Domain", Value = _settings.Domain });
s.Add(new EmailToken
{
Token = "Subject",
Value =
string.Format("New User {0} has joined {1} on myGrover.", savedUser.FullName(), agency.Name)
});
s.Add(new EmailToken { Token = "Name", Value = savedUser.FullName() });
return s;
});
}
}
}
Let’s examine the testability of the above code.
Is this simple code?
Yes, it is, the cyclomatic complexity is below 5.
Are there any dependencies?
Yes. There are 2 services AgencySettingsRepository and EmailService.
Are the services mockable?
No, their creation is hidden within the method.
Is the code testable?
No, this code isn’t testable because we can’t mock AgencySettingsRepository and EmailService.
How can we make this code testable?
We inject (using constuctor injection) AgencySettingsRepository and EmailService as dependencies. This allows us to mock them from a test and test in isolation.
Below is the refactored version.
Notice how the services are injected into the constructor. This allows us to control which implementation is passed into the SendMail constructor. It’s then easy to pass dummy data and intercept the service method calls.
public class SendEmail
{
private IAgencySettingsRepository _agencySettingsRepository;
private IEmailService _emailService;
public SendEmail(IAgencySettingsRepository agencySettingsRepository, IEmailService emailService)
{
_agencySettingsRepository = agencySettingsRepository;
_emailService = emailService;
}
public void SendUserHadJoinedEmailToAdministrator(DataAccess.Database.Schema.dbo.Agency agency, User savedUser)
{
var agencySettings = _agencySettingsRepository.GetById(agency.Id);
if (agencySettings != null)
{
var newAuthAdmin = agencySettings.NewUserAuthorizationContact;
if (newAuthAdmin.IsNotNull())
{
_emailService.SendTemplate(new[] { newAuthAdmin.Email },
GroverConstants.EmailTemplate.NewUserAdminNotification, s =>
{
s.Add(new EmailToken { Token = "Domain", Value = _settings.Domain });
s.Add(new EmailToken
{
Token = "Subject",
Value = string.Format("New User {0} has joined {1} on myGrover.", savedUser.FullName(), agency.Name)
});
s.Add(new EmailToken { Token = "Name", Value = savedUser.FullName() });
return s;
});
}
}
}
}
Below is an example of testing in isolation. We are using the mocking framework FakeItEasy.
[Test]
public void TestEmailService()
{
//Given
//Using FakeItEasy mocking framework
var repository = A<IAgencySettingsRepository>.Fake();
var service = A<IEmailService>.Fake();
var agency = new Agency { Name = "Acme Inc." };
var user = new User { FirstName = "Chuck", LastName = "Conway", Email = "chuck.conway@fakedomain.com" }
//When
var sendEmail = new SendEmail(repository, service);
sendEmail.SendUserHadJoinedEmailToAdministrator(agency, user);
//Then
//An exception is thrown when this is not called.
A.CallTo(() => service.SendTemplate(A<Agency>.Ignore, A<User>.Ignore)).MustHaveHappened();
}
Writing defect resistance code is surprisingly easy. Don’t get me wrong, you’ll never write defect-free code (if you figure out how, let me know!), but by following the 4 practices outlined in this article you’ll see a decrease in defects found in your code.
To recap, Writing Simple Code is keeping the cyclomatic complexity around 5 and the method size small. Writing Testable Code is easily achieved when following the Inversion of Control and the S.O.L.I.D Principles. Code Reviews help you and the team understand the domain and the code you’ve written — just having to explain your code will reveal issues. And lastly, Unit Testing can drastically improve your code quality and provide documentation for future developers.
In software engineering, there is a prevailing idea that an engineer should only use a framework when he or she understands the internal workings. This is a fallacy.
Why is it that we must know the internal workings — do the details matter that much? Some might say ignorance is bliss.
Let’s examine the engine of a car:
How many really know how the engine works?
Can you tell me why it’s called a 4 stroke engine?
What does each stroke do?
What’s the difference between a 4 stroke engine and a 2 stroke engine?
Anyone?
And yet we still drive our cars without any thought on “how” the car is getting us to our destination.
We interface with the car using the steering wheel, the gear shifter, the gas pedal, and the brakes.
Who cares how it works, as long as it gets us to our destination. When the car breaks down we take it to an expert.
In business, a company has specialized knowledge that allows it to be competitive. This is referred to as a company’s core competency.
A core competency can be a process or a product.
To stay competitive, a company must tirelessly improve their core competency. Using resources for activities other than supporting the company’s core competency weakens the company’s competitive advantage. Which opens the window of opportunity for competitors to overtake the company’s competitive advantage.
This idea is best illustrated with an example.
Apple is known for their simplicity and their beautiful products. You’d think this would be easy to replicate, but it’s not, just ask Samsung, HTC, and Microsoft.
Why have these companies failed? Because simple is hard and Apple is expert in simple.
Core competency can apply to people too.
What sets you apart from others?
To have developed your core competency, you’ve had to rigorously focus in one area, sometime for years, gaining insights and knowledge setting you apart from others.
As in a business, to maintain your competitive advantage you must continually hone your core competency.
A software engineer is no different from a company or any other professional. We must pick and choose what we learn to stay aligned with our core competency.
Understanding the internals of every framework we use is not practical and is time consuming. I’m expecting the framework’s author to be an expert in the framework’s domain, therefore, I don’t need to know it’s internal workings.
Isn’t this the point of software — to use black boxed bits of functionality to produce a larger more complex work? I believe it is.
In the end, it comes down to focus and time, both of which are limited.
Everyone has a favorite editor. We each have reasons for choosing our editor. I’ve tried them all. And I’ve found that Brackets.io best suits me. Unfortunately, there are gaps in the functionality of Brackets.io. With a robust ecosystem of extensions, I’ve found 8 extensions that complete Brackets.io.
Here is a list of my 8 must have extensions.
For anyone working with CSS and HTML Emmet is a must have. I wrote about it earlier this year. It removes all the unnecessary typing from while create HTML and CSS.
It’s officially called “Autosave Files on Window Blur”. This extension saves all the changes files once you’ve navigated away from Brackets. It works similarly to how WebStorm saves it’s files.
You’d think this wasn’t a big deal, at least that’s what I thought. But it does a great job! Give it a try. You’ll be surprised how useful this plugin is — Beautify
This is the best git integration I have ever used. And I’ve used Git in WebStorm, Sublime Text and Visual Studio. So that’s saying a lot. It’s functional and aesthetically pleasing, there isn’t much else to ask for. – Brackets Git
You’d be surprised how much a few good icons can spruce up an ole editor. – Brackets Icons
In my opinion this is a missing feature of Brackets.io. This completes the editor. – Documents Toolbar
This summarizes all the TODO comments in the file. It also supports NOTE, FIXME, CHANGES and FUTURE. More can be added if this list is too limiting. – Todo
This extension automatically highlights occurrences of the selected word. Much like Notepad++ and Sublime Text. – Quick Search
I found the lack of right-click cut and paste annoying. In Windows, right-click cut and paste is bread and butter of my workflow. Out of the box Brackets is missing the right-click cut and paste functionality. This extension saves the day by adding it. – Right-Click Extended
I went through blood, sweat and tears to bring this to you. I suffered the scorching heat of Death Valley and summited the peaks of Mount McKinley. I’ve sacrificed much.
Much of content shared in the post is not my original work. Where I can, I link back to the original work.
This article assumes you can get around Linux.
I could not find a comprehensive guide on hosting and managing Nodejs applications on Ubuntu in a production capacity. I’ve pulled together multiple articles on the subject. By the end of this article I hope you’ll be able to setup up your own Ubuntu server and have Nodejs deploying via a continuous integration server.
I am using TeamCity on Windows which then deploys code from GitHub to Ubuntu hosted on AWS.
For this article I used the following technologies:
I’m not going into detail here. Amazon Web Services (AWS) makes this pretty easy to do. It doesn’t matter where it’s at or if it’s on your own server.
I encountered a few gotchas. First, make sure port 80 is opened. I made the foolish mistake of trying to connect with port 80 closed. Once I discovered my mistake, I felt like a rhinoceros’s ass.
Nodejs is a server technology using Google’s V8 javascript engine. Since it’s release in 2010, its become widely popular.
The following instructions originally came from a Digital Ocean post).
You always have the option to install Nodejs from the apt-get, but it will be a few versions behind. To get the latest bits, install Nodejs from the source.
At this send of this section we will have downloaded the latest stable version of node (as of this article), we will have build the source and installed Nodejs.
Log into your server. We’ll start by updating the package lists.
sudo apt-get update
I’m also suggesting that you upgrade all the packages. This is not necessary, for Nodejs but it is good practice to keep your server updated.
sudo apt-get upgrade
Your server is all up to date. It’s time download the source.
cd ~
As of the writing 12.7 is the latest stable release of Nodejs. Check out nodejs.org for the latest version.
wget https://nodejs.org/dist/v0.12.7/node-v0.12.7.tar.gz
Extract the archive you’ve downloaded.
tar xvf node-v*
Move into the newly created directory
cd node-v*
Configure and build Nodejs.
./configure
make
Install Nodejs
sudo make install
To remove the downloaded and the extracted files. Of course, this is optional.
cd ~
rm -rf node-v*
Congrats! Nodejs is now installed! And it wasn’t very hard.
Nodejs can act as a web server, but it’s not what I would want to expose to the world. An industrial, harden, feature rich web server is better suited for this. I’ve turned to Nginx for this task.
It’s a mature web server with the features we need. To run more than one instance of Nodejs, we’ll need to port forwarding.
You might be thinking, why do we need more than one instance of Nodejs running at the same-time. That’s a fair question… In my scenario, I have one server and I need to run DEV, QA and PROD on the same machine. Yeah, I know not ideal, but I don’t want to stand up 3 servers for each environment.
To start let’s install Nginx
sudo -s
add-apt-repository ppa:nginx/stable
apt-get update
apt-get install nginx
Once Nginx is has successfully installed we need to set up on the domains. I’m going to assume you’ll want to have each of your sites on it’s own domain/sub domain. If you don’t and want to use different sub-folders, that’s doable and very easy to do. I am not going to cover that scenario here. There is a ton of documentation on how to do that. There is very little documentation on setting up different domains and port forwarding to the corresponding Nodejs instances. This is what I’ll be covering.
Now that Nginx is installed, create a file for yourdomain.com at /etc/nginx/sites-available/
sudo nano /etc/nginx/sites-available/yourdomain.com
Add the following configuration to your newly created file
# the IP(s) on which your node server is running. I chose port 9001.
upstream app_myapp1 {
server 127.0.0.1:9001;
keepalive 8;
}
# the nginx server instance
server {
listen 80;
server_name yourdomain.com;
access_log /var/log/nginx/yourdomain.log;
# pass the request to the node.js server with the correct headers
# and much more can be added, see nginx config options
location / {
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://app_myapp1;
}
}
Make sure you replace “yourdomain.com” with your actual domain. Save and exit your editor.
Create a symbolic link to this file in the sites-enabled directory.
cd /etc/nginx/sites-enabled/
ln -s /etc/nginx/sites-available/yourdomain.com yourdomain.com
To test everything is working correctly, create a simple node app and save it to /var/www/yourdomain.com/app.js and run it.
Here is a simple nodejs app if you don’t have one handy.
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');}).listen(9001, "127.0.0.1");
console.log('Server running at http://127.0.0.1:9001/');
Let’s restart Nginx.
sudo /etc/init.d/nginx restart
Don’t forget to start your Nodejs instance, if you haven’t already.
cd /var/www/yourdomain/ && node app.js
If all is working correctly, when you navigate to yourdomain.com you’ll see “Hello World.”
To add another domain for a different Nodejs instance your need to repeat the steps above. Specifically you’ll need to change the upstream name, the port and the domain in your new Nginx config file. The proxy_pass address must match the upstream name in the nginx config file. Look at the upstream name and the proxy_pass value and you’ll see what I mean.
To recap, we’ve installed NodeJS from source and we just finished installing Nginx. We’ve configured and tested port forwarding with Nginx and Nodejs
You might be asking “What is PM2?” as I did when I first heard about. PM2 is a process manager for Nodejs applications. Nodejs doesn’t come with much. This is part of it’s appeal. The downside to this, is well, you have to provide the layers in front of it. PM2 is one of those layers.
PM2 manages the life of the Nodejs process. When it’s terminated, PM2 restarts it. When the server reboots PM2 restarts all the Nodejs processes for you. It also has extensive development lifecycle process. We won’t be covering this aspect of PM2. I encourage you to read well written documentation.
Assuming you are logged into the terminal, we’ll start by installing PM2 via NPM. Npm is Nodejs package manager (npm). It was installed when you installed Nodejs.
sudo npm install pm2 -g
That’s it. PM2 is now installed.
PM2 is easy to use.
The hello world for PM2 is simple.
pm2 start hello.js
This adds your application to PM2’s process list. This list is output each time an application is started.
In this example there are two Nodejs applications running. One called api.dev and api.pre.
PM2 automatically assigns the name of the app to the “App name” in the list.
Out of the box, PM2 does not configure itself to startup when the server restarts. The command is different for the different flavors of Linux. I’m running on Ubuntu, so I’ll execute the Ubuntu command.
pm2 start ubuntu
We are not quite done yet. We have to add a path to the PM2 binary. Fortunately, the output of the previous command tells us how to do that.
Output:
[PM2] You have to run this command as root
[PM2] Execute the following command :
[PM2] sudo env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u sammy
Run the command that was generated (similar to the highlighted output above) to set PM2 up to start on boot (use the command from your own output):
sudo env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u sammy
Examples of other PM2 usages (optional)
Stopping an application by the app name
pm2 stop example
Restarting by the app name
pm2 restart example
List of current applications managed by PM2
pm2 list
Specifying a name when starting a process. If you call, PM2 uses the javascript file as the name. This might not work for you. Here’s how to specify the name.
pm2 start www.js --name api.pre
That should be enough to get you going with PM2. To learn more about PM2’s capabilities, visit the GitHub Repo.
You are probably thinking, “What in the name of Betsey’s cow is Plink?” At least that’s thought. I’m still not sure what to think of it. I’ve never seen anything like it.
You ever watched the movie Wall-e? Wall-e pulls out a spork. First he tries to put it with the forks, but it doesn’t fix and then he tries to put it with the spoons, but it doesn’t fit. Well that’s Plink. It’s a cross between Putty (SSH) and the Windows Command Line.
Plink basically allows you to run bash commands via the Windows command line while logged into a Linux (and probably Unix) shell.
Start by downloading Plink. It’s just an executable. I recommend putting it in C:/Program Files (x86)/Plink. We’ll need to reference it later.
If you are running an Ubuntu instance in AWS. You’ll already have a cert setup for Putty (I’m assuming you are using Putty).
If you are not, you’ll need to ensure you have a compatible ssh cert for Ubuntu in AWS.
If you are not using AWS, you can specify the username and password in the command line and won’t to worry about the ssh certs.
Here is an example command line that connects to Ubuntu with Plink.
"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk"
This might be getting ahead of ourselves, but to run an ssh script on the Ubuntu server we add the complete path to the end of the Plink command.
"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk" /var/www/deploy-dev-ui.sh
And that, dear reader, is Plink.
NODE_ENV is an environment variable made popular by expressjs. Before your start the node instance, set the NODE_ENV to the environment. In the code you can load specific files based on the environment.
Setting NODE_ENV
Linux & Mac: export NODE_ENV=PROD
Windows: set NODE_ENV=PROD
The environment variable is retrieved inside a Nodejs instance by using process.env.NODE_ENV.
example
var environment = process.env.NODE_ENV
or with expressjs
app.get('env')
*Note: app.get(‘env’) defaults to “development”.
Nodejs, PM2, Nginx and Plink are installed and hopefully working. We now need to bring all these pieces together into a continuous integration solution.
Clone your GitHub repository in /var/www/yourdomain.com. Although SSH is more secure than HTTPS, I recommend using HTTPS. I know this isn’t ideal, but I couldn’t get Plink working with GitHub on Ubuntu. Without going into too much detail Plink and GitHub SSH cert formats are different and calling GitHub via Plink through SSH didn’t work. If you can figure out the issue let me know!
To make the GitHub pull handsfree, the username and password will need to be a part of the origin url.
Here’s how you set your origin url. Of course you’ll need to substitute your information where appropriate.
git remote set-url origin https://username:password@github.com/username/yourdomain.git
Clone your repository.
cd /var/www/yourdomain.com
git clone https://username:password@github.com/username/yourdomain.git .
Note, that if this directory is not completely empty, including hidden files Git will not clone the repo to this directory.
To find hidden files in the directory run this command
ls -a
For the glue, we are using a shell script. Here is a copy of my script.
#!/bin/bash
echo "> Current PM2 Apps"
pm2 list
echo "> Stopping running API"
pm2 stop api.dev
echo "> Set Environment variable."
export NODE_ENV=DEV
echo "> Changing directory to dev.momentz.com."
cd /var/www/yourdomain.com
echo "> Listing the contents of the directory."
ls -a
echo "> Remove untracked directories in addition to untracked files."
git clean -f -d
echo "> Pull updates from Github."
git pull
echo "> Install npm updates."
sudo npm install
echo "> Transpile the ECMAScript 2015 code"
gulp babel
echo "> Restart the API"
pm2 start transpiled/www.js --name api.dev
echo "> List folder directories"
ls -a
echo "> All done."
I launch this shell script with TeamCity, but you can launch with anything.
Here is the raw command.
"C:\Program Files (x86)\Plink\plink.exe" -ssh ubuntu@xx.xx.xx.xx -i "C:\Program Files (x86)\Plink\ssh certs\aws-ubuntu.ppk" /var/www/deploy-yourdomain.sh
exit
>&2
That’s it.
This process has some rough edges… I hope to polish those edges in time. If you have suggestions please leave them in the comments.
This document is in my GitHub Repository. Technologies change, so if you find an error please update it. I will then update this post.
Have patience.
Coding is discovery. Coding is failing. Be ok with this.
Don’t blame the framework. It’s more probable it’s your code. Accept this fallibility.
Know when to walk away. You mind is a wonderful tool, even at rest it’s working on unsolved problems. Rest, and let your mind do it’s work.
Be comfortable not knowing. Software engineering is a vast ocean of knowledge. Someone will always know more than you. The sooner you are OK with this the sooner you will recognize the opportunity to learn something new.
Anger and frustration don’t fix code. Take a break, nothing can be accomplished in this state.
