If you're building out the technical landscape of your startup, take a look at Kate Matsudaira's excellent post yesterday about "Assessing Technical Risks for Startups – New Tech Leader Series".

Having gone through technical due diligence twice in my career (both of which involved the sale of a startup to a large company, the first being my ISP, Pacific Rim Network to the ISP Verio, and the second being Classmates.com to now-parent United Online), there are a few items I'd add to Kate's excellent list.

My original list included 72 items, using slightly different categories than Kate's. Rather than reproducing the entire list (with duplicates) here are the items on my list not explicitly on hers. (Admittedly, her list was more for familiarizing yourself with a new environment, but this is partly the aim of technical due diligence as well, so both her and my list are cross functional that way).

Engineering Focus Items

• What does the development environment look like?
• Developer tools, desktops, environments, data should be reasonably similar to production environments to ensure no environmental defects are introduced to production.
• What's the Code Review Policy?
• Engineering should have stricter controls and peer review in place for sensitive areas of the codebase. In particular, anything you do with credit card processing, or storing private, sensitive or legally protected data should have a formal peer review process in place. Even lacking these higher requirements, embracing such a process has been a hallmark of high-performing teams I have managed.
• Software Lifecycle management strategy?
• The code an organization relies on to run its business is akin to the food pack a travelling nomad brings along for the journey-- everything ought to be as fresh as possible, and that mystery meat should be thrown out, stat! The Y2K problem was arguably greatest for those organizations who had tons of COBOL code running their production systems without any COBOL programmers on staff. You should be asking "who owns our oldest code, when was the last time it was touched, and do we need it anymore?" Every line of code in your source code system should be adding to shareholder value, and the best place to start your analysis is typically your oldest codebase.
• Documentation, documentation, documentation.
• This is on Kate's list, but it bears expanding upon. Not only should appropriate documentation (akin to javadocs) be in-line in your source code, any artifacts that were instrumental in software design should live in a central repository (I prefer a well-organized wiki), and the culture of the organization should ensure that these artifacts are used, and re-used, or pruned when they are no longer relevant. Ideally this culture extends beyond engineering to product development, marketing and beyond.
• Security
• Are there periodic security training sessions? When was the last one held? Did it cover the OWASP Top Ten? Are the software engineers familiar with the principal attack vectors your application is potentially vulnerable to? Are your testers aware of various techniques to use to validate that your products are not vulnerable?
• Testing Strategy?
• Is testing a hodge-podge of "what feels right", or is there a strategic viewpoint? For example, are your libraries and low level code unit tested for a reason, or just by accident? Do your higher-level systems use integration and build verification tests? Are these tests automated? How often does the automation run? Do failures break the build?
• Technical Contingency Planning
• What strategies have been used until now to keep pace with scalability? What's in progress, and what does the future look like? What tier has the greatest pain? What does growth look like? Linear, logarithmic, exponential? What's the doubling speed? Are infrastructure cost needs reflected in the budget?
• Caching Strategy
• Is the architecture demonstrably using every caching tier effectively? (Browser, Edge, Web, Application, Database)

Configuration Management/Deployment Focus Items

• What is the Deployment & Configuration Management Strategy?
• 3rd party and vendor software is often deployed using RPMs, or using package managers like yum, apt, or (heaven forbid) tarballs or system-based software package managers. But what do you use for internally built software? These tools are often language specific (ant/maven for java, for example, or gems for ruby, or pear for php), and in hybrid environments, having engineers package software releases into rpms shows a higher degree of aptitude (pardon the pun) in configuration management and deployment.

Information Technology (IT) Focus Items

• 3rd Party Licenses
• All 3rd party software on engineers' desktops should be properly licensed. A culture of respecting others' intellectual property must be encouraged, and a process that directs new software requests to management (for consideration whether single-, multiple- or a site- licenses should be acquired).
• Proper ownership of technical assets?
• Are ownership of technical assets in the right hands? From domain name registrations to licenses to support, telco and datacenter contracts, they should, to the extent possible, be using role-based accounts monitored by multiple people, or if directed at a person, ideally someone high up enough in the chain to have fiduciary responsibility.
• Customer Support System
• What's the ticketing system?
• Policy Compliance
• This is a very broad area. To the extent it is relevant, are your software products in compliance with regulations including (but certainly not limited to) CAN-SPAM, Privacy/Terms of Service, COPPA, PCI-DSS, SOX, HIPAA, etc.?
• Hardware Documentation
• Do you have a list of all of your hardware assets? When were they acquired? When are they due to be replaced? Are they granular enough to take into account sub-systems? (Cards, hard drives) What software versions do they run (i.e. are they firmware and software patched and up to date)?
• Legacy Assessment
• What systems, software and hardware are considered legacy? (To me, this means, not currently under an active support contract, end-of-lifed, no bugfix policy, or hardware older than 3 years old). Are they that way because of certain software systems, or vice versa? What's the plan for untangling the mess?

Operations Focus Items

• Network Bandwidth
• What's current base and peak utilization? If peak is approaching > 30%, what are the plans on what timeline to expand bandwidth needs? (This applies to every central or branch office, as well as to any critical or backup datacenter, or to any VPNs or leased circuits)
• Single Points of Failure
• Kate touches on this from a broad perspective, but it bears considering network path redundancy. What happens if a given link is cut? Say a backhoe takes out a bundle of fiber. What alternatives for every business-critical system do you have in place?
• Hosting Infrastructure Review
• Kate also touched a lot on this area, but ensure that your DNS, Web, Database, Data Warehouse, Reporting and Monitoring systems have no single points of failure, and if they do, a mitigation, disaster recovery, remediation or response plan should be identified.
• Escalation Lists
• You should have a list of escalation contacts, technical, business or otherwise, with every 3rd party service provider, from facilities (your office building, data center) to transit provider (telco/ISP) to service providers (cloud hosting, cloud service provider, etc.). Ideally copies of these for Technology would be kept in the same place as your SLAs and contracts.
• Infrastructure Concerns
• These are mostly environmental. Space, power & cooling (both nominal and backup) in your various places of business, particularly colo/hosting facilities.
• System Administration Policies
• In line with PCI-DSS, do you have controls (audits) of who did what to what system? Who has administrative access (this list should generally be constrained to key operations folks). Who has the root password (this list should be ultra-minimal, allowing most administrators to use sudo or passwordless ssh-to-root style access)?
• Environment Restrictions?
• Do you enforce environment restrictions ensuring that only people who "need" access have it to any given environment?
• Application Restrictions?
• Whether it's a bug tracking system, ad server system, customer support system or otherwise, ideally access to these systems are governed by centralized policy, authentication and controls (say, LDAP for example).
• Physical Security
• Are keys (physical or digital) to critical systems locked up, with backups, with restricted access? Are critical systems secured physically and digitally?
• Backup Access?
• Access to backups should be treated as sensitively as the most sensitive data they contain. Archives of credit card numbers should be under lock and key in a bolted-down safe, whereas HTTP access logs can just be locked up in an office, for example.
• Data Retention Policy
• Is it defined, and adhered to?

I hope this list, in conjunction with Kate's, helps you identify what opportunities your startup faces to raise the technical diligence of your organization. If there's something missing from our lists, please add it to the comments below!

This post is intended mostly to benefit various searchers with a relatively obscure but frustrating technical problem, so apologies to my regular blog readers if this is not directly relevant to your interests.

If you're still reading, it's probably because you're trying to set up DKIM, or more specifically dkim-milter/dkim-filter on Postfix or Sendmail or something like it. I struggled with this issue on and off for a couple days. The biggest obstacle ended up being seeing logs like these in my maillog:
dkim-filter [...] SSL error:0906D06C:PEM routines:PEM_read_bio:no start line dkim-filter [...] dkim_eom(): resource unavailable: PEM_read_bio_PrivateKey() failed

After a ton of searching online, the issue pointed to a formatting error in the certificate I created using a dkim-genkey.sh script in an article that explained how to set it up. Specifically, it created a certificate file that lacked the -----BEGIN CERTIFICATE----- style header/footer. I tried manually adding these back in, re-creating the line breaks to build a standard .pem formatted file, but none of these seemed to work.

Ultimately, the way I fixed it was to use the "dkim-genkey -d [domain name]" command that came with the dkim distribution to generate the key and DKIM DNS record.

After restarting dkim-milter and postfix, and sending a test message to check-auth@verifier.port25.com (domain names munged below, but otherwise this is cut/paste from my unix cli)...
$ telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 mail.example.com ESMTP Postfix MAIL FROM: khan@example.com 250 2.1.0 Ok RCPT TO: check-auth@verifier.port25.com 250 2.1.5 Ok DATA 354 End data with . SUBJECT: Testing Test . 250 2.0.0 Ok: queued as 9A30E76C137 QUIT 221 2.0.0 Bye Connection closed by foreign host.

Here's the response I got:

[snip] ---------------------------------------------------------- DKIM check details: ---------------------------------------------------------- Result: pass (matches From: khan@example.com) ID(s) verified: header.d=example.com Canonicalized Headers: Subject:'20'Test'0D''0A' To:'20'check-auth@verifier.port25.com'0D''0A' Message-Id:'20'<20120411204224.3839576C138@www.example.com>'0D''0A' Date:'20'Wed,'20'11'20'Apr'20'2012'20'13:42:13'20'-0700'20'(PDT)'0D''0A' From:'20'khan@example.com'0D''0A' DKIM-Signature:'20'v=1;'20'a=rsa-sha1;'20'c=simple/simple;'20'd=example.com;'20's=default;'0D''0A' '09't=1334176963;'20'bh=pN6k1Y/YJkrV9QxvE85uD6qiAqw=;'0D''0A' '09'h=Subject:To:Message-Id:Date:From;'0D''0A' '09'b= Canonicalized Body: Testing'0D''0A' DNS record(s): default._domainkey.example.com. 1800 IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDX3yuDnEl5XL2FPPKxW5h9iW43ZMxlg2hI/23BiYXczMFokP7AjsNMZKzg/9s1E2TswBFMFnM26ZDQtV7P3FmlcRO2H3YmRfIOftY 93+c88KXKN1fET5PfyKjWqraYHJiNID+Vwzn5njDfI1WnDxkzVPEx/30v7W3tfbCpAJGhkwIDAQAB"

In Part I, I outlined why it makes sense to focus on your people as the foundation of a technology management strategy.

In the People/Process/Technology construct, I believe that process, particularly process that is aligned with empowering, organizing, and adapting to eliminate the impediments of your people is the next critical aspect that needs attention.

Increasingly, and certainly for startups whose principal value is captured in the intellectual capacity or capital of its people, effective development and organizational process is critical to applying that brainpower to the problems that your company is out there trying to solve.

So, ask yourself, are you maximizing the intellectual abilities of your team? Are you engaging and challenging them to contribute, or are they being invited to stop thinking and just do what's handed down from on high?

Are you at risk of having your policies and procedures resulting in having your employees stop thinking, stop trying to find a better way, and to stop asking insightful questions? If you’re using process to introduce handcuffs, consider why?

There may be good reasons to institute oppressive or onerous policy decisions. Maybe you've inherited or hired the wrong people. Or you can't or shouldn't trust your people to know well enough to do the right thing. Or perhaps the risks you face too are great (from a regulatory, legal, or economic perspective)?

Even for the most conservative of organizations, these elements can be changed as part of company culture through proper one-on-one mentoring, encouraging empowered teams, and in drastic situations, through personnel changes.

Ultimately, I believe every technology manager's goal can be summed up as the "three E's":

• Educate (your people)
• Eliminate (their impediments)
• Emancipate (Set them free to do the same across your organization)

There are a lot of ways to do this, and I don't want to be evangelical about how this must come about. Some of you will be able to do it in waterfall organizations. Personally, I have found that embracing Agile methodologies (in my case, Scrum) both adheres to the right things, as well as introduces a significant enough change to warrant re-evaluation by your team members of "the way things are and if things should continue in that way".

Inspired by the LEAN school of management pioneered by Toyota, Agile methodologies like Scrum and Kanban have evolved the art of software engineering to be more predictable, and towards more collaborative team-oriented development on short iterations called “sprints”.

In a nutshell, and in contrast to the traditional "waterfall" approach of development, instead of long, drawn-out, and unpredictable development death-marches, where the deliverables are big payloads with big risk, agile adopts 1-3 week dev iterations, that tend to represent much more knowable, much more manageable cycles.

Because cycles happen quickly, teams have a lot of time to practice design, development, testing, integration, deployment. In my organization this happens nearly every single week, meaning we are challenged constantly to improve the status quo in all of these dimensions.

You've heard the adages that "practice makes perfect". I've been in an organization where launching a product took upwards of a year. If your launch team only launches one major product in one year, are we surprised that they get rusty in developing code that is production ready? On the other hand, if your team launches code to production dozens of times a year, who's more likely going to be able to launch dozens of times a day?

Long before Agile and Scrum were even coined as development methodologies, Tom DeMarco and Timothy Lister in the excellent book "Peopleware, Productive Projects & Teams" summarized one of the benefits of iterative development:

“The chemistry-building manager takes pains to divide the work into pieces and makes sure that each piece has some substantive demonstration of its own completion.

Such a manager may contrive to deliver a product in twenty versions, even though two are sufficient for upper management and the user...

Each new version is an opportunity for closure. Team members get warmed up as the moment approaches, they sprint near the very end. They get a high from success. It suffuses them with renewed energy for the next step. It makes them feel closer together.”

Tom DeMarco & Timothy Lister, PeopleWare, 1987

Beyond the concept of "sprints" in agile development methodologies, Agile advocates having clear strategic vision from the top of the organization, to "stratical" mid-term roadmap planning, to the tactical mechanics of day to day standups and sprint planning meetings.

The most critical aspect to agile methodologies, in my humble opinion, however, is the power of the retrospective. Not only does Agile focus on removal of impediments, tackling problems collaboratively, and providing transparency to the challenges and triumphs of tackling your commitments, but it provides an avenue for people to express endless curiosity and continuous team and self improvement.

My friends have coined a term called "Khanalogy" for my penchant to use analogies to explain sometimes mundane concepts. So I'll share a Khanalogy with you now...

What would you rather be-- a business organism that has no auto-immune response... an organism that can't recognize threats to its resources, its capabilities and its long-term survival.... Or an organism that recognizes when things aren't going well? One that detects problems, assumes responsibility for fixing them, asks for help when required, and brings the appropriate (and sometimes full) force of that organization's prowess to utterly decimating the impediments that afflict that organism?

Do you want to micromanage this behavior, like a drill sergeant rousing its unfit cadets out of their slumber to react to a threat, or do you want an autonomous system that can spring into action like an elite special ops team like the secret service protecting the president?

If your team isn't full of people who are motivated, empowered, and willing to engage the problems and impediments preventing them from becoming a world-class engineering organization, properly implemented Agile methodologies may represent a set of tools that could kick that into high gear.

In Part 3, I tackle the technology dimension of the trilemma.

I fundamentally believe that we live more fulfilled personal and professional lives when we apply our strengths to where they matter most.

Start By Discarding the "Meh"

Mac or PC? Republican or Democrat? Vanilla or Chocolate?

For many people, they have instant reactions to questions like these. I'm here to advocate getting off the fence and having an opinion, even at the risk of being wrong.

One of my personal heroes, Walt Disney, was asked how he kept persevering in the face of difficulty.

"We keep moving forward, opening up new doors and doing new things, because we're curious... and curiosity keeps leading us down new paths."

And Walt Disney, like another of my personal heroes, Steve Jobs, was an opinionated guy. You don't build products and services that have the effect of a religious experience saying "Meh, I guess".

And some of you might be bristling at my characterizations above. "What about Linux?" "I don't care as long as its gelato." "Well my political leanings are better explained as a mix of Jeffersonian democracy and Rousseauian humanism". That's what I'm talking about. Enlightened opinions are entrenched, often not intractably, from a given perspective which leads to insights and observations that challenge the status quo.

And it is this challenge of the status quo where I believe the greatest insights (and not accidentally, the greatest value!) is often generated. Discard the illusion that "it doesn't matter". The only time it doesn't really matter is when you don't care. When you care, and when you can find and work with people who have an equal and honest intellectual honesty to what they care about, you can begin to have conversations and revelations about the topic of debate that are truly enlightening.

My Strongly Held Opinion: People First

One construct in technology management that I've been posed is the "Trilemma" or "Iron Triangle" of People/Process/Technology. It's a false dichotomy, since in reality, it's never quite this black-or-white, but the question is often asked like this: Your boss informs you that of the following three areas, you may keep 70% of one, 20% of another, and 10% of the remainder... how would you choose if you had to decide between People, Process, or Technology?

For example, if you had your staff, Agile development/training resources, and some fancy technology (Ruby, Erlang, Haskell, Pascal, Fortran99, pick your favorite), how would you choose? If you pick Technology (70%) and Process (20%) your boss tells you that the team working with that technology and process is some outsourced dev team half way around the globe you didn't get to screen, interview, or hire.

For me, it helps to view this triangle as a pyramid, whose foundation is people. In a nutshell, the logic works like this: Technology is ever changing, and if there's a universal constant, it's that yesterday's technology sucks worse than today's. And in retrospect what you're working on right now, as cool as it may seem, is going to be obsolete, and need to be replaced, updated, refactored, redesigned, and rebuilt. Time and progress has a way of obsoleting the best laid plans, and as we all know fairly well, the best laid plans are often made with incomplete, inaccurate, or uncertain data.

Even if you consider yourself well-applied in your given areas of expertise, clearly you weren't always so. What has allowed you to distinguish yourself in your given areas of expertise is a desire to learn-- a desire to apply yourself. And aren't these more important aspects of what you have to offer than your current knowledge?

Your ability to integrate new learnings, new experiences and new techniques to the problem at hand is arguably more valuable than the sum total of all the book knowledge you currently possess.

And when you consider how much more productive your years of experience makes you today compared to where you were a decade or two ago, this gets compounded with time, while technologies fall by the wayside.

A similar argument can be made for the People vs. Process argument. Every process, every policy, every rule requires people to execute them. With "The Right Stuff" on your team, you could probably make a good pass at your software projects with Waterfall, RUP, Scrum, Kanban, or just plain "log it in Bugzilla/Redmine and we'll get to it when we get to it".

The Bottom Line

False dichotomies are a type of logical fallacy, and I'm not suggesting that any of the real choices you face will ever be as stark to have the iron triangle above shed critical light on them. But it does help to cement your worldview-- that if you had to allocate resources, strategies, and focusing on the things that matter the most, you're best off focusing first on your people for they are the foundation upon which everything else you build is built.

And more directly, it's about the kind of people you want to hire. I helped write the "Careers" page for Salad Labs, the company I work for, and as we state,

"We fill jobs by looking for people who are the best equipped to solve certain types of problems over the course of months and years...

We believe that smart, motivated, creative people can learn just about anything, and quickly. When hiring, we try to err on the side of smart...

This is an entrepreneurial venture, and we look for people with an entrepreneurial spirit — people who are willing to be accountable, take the lead, take measured risks, and get things done. Note that, in this context, “entrepreneurial spirit” is not code for “does not play well with others.” We work as a team, and we hire nice people."

A couple months ago, I gave a talk at EmMeCon on a topic near and dear to my heart- where the trends in technology are going, and what the astute technical manager needs to be aware of to stay competitive in the ever-changing landscape.

I've reworked the outline of that talk in to a five-part essay, and am sharing it with you on my blog. My tentative outline looks like this:

• Part I: Maximize the Value of Your People
• Part II: Making the Most of your Process
• Part III: Making the Most of your Technology - The Historical Perspective
• Part IV: Making the Most of your Technology - Go with the Flow
• Part V: Making the Most of your Technology - A Few Words of Warning

I'll also follow up with a few thoughts that got cut from the talk since it was peripheral and off-topic to the main point of the talk, but that work just fine in the context of a blog.

Stay tuned for Part I!