Open Source Solutions for Small Business Problems

The Linux Link Tech Show (TLLTS) has a great segment dissecting the criticisms/wild flames put forth on a series of shows on the TWIT network. Wanted to add a couple comments missing from their discussion.

First of all, the Mac Break Weekly show apparently spends some time bashing the open source community, calling out Drupal, and how difficult it is to solve “simple” problems like uploading images for blog posts. In practically the same breath, the hosts claim that the open source community never has any innovation behind it. Irony drips:

• In the world of content management systems (CMSs), most of the innovation starts in open source projects these days, and Drupal is at the cutting edge of this with its powerful system of taxonomies, and hundreds of add-ons freely available.
• I can think of a grand total of 2 proprietary CMSs that have anywhere near as widespread use as most of the open source CMS systems. One has turned open source itself: Movable Type. The other is Sharepoint, and it’s widespread because Microsoft has shipped it out on lots of its server products.
• Complain about usability all you want, but name a proprietary product as powerful as Drupal, that’s easier to install, administer, and configure.
• The TWIT.tv site itself is running on Drupal.

So let’s talk a little about innovation. While Photoshop may still dominate the world of graphic design, but the lines aren’t so clear when it comes to animation. The Blender project recently released its second short animated film, Big Buck Bunny. While you might argue about the strength of its story, you cannot deny that the technical effects are as stunning as any major animated film coming out from the big studios. And it was created by 7 people in 9 months, using open source software. Even the big studios like Pixar, Dreamworks, and Industrial Light and Magic rely on open source software to deliver their magic, such as CinePaint, POVray, and several others.

On the subject of innovation, KDE4 is breaking new ground and stirring up controversy, laying a bedrock that promises the ability to do things beyond the standard “Desktop” paradigm that was invented over 30 years ago and we’ve all used ever since. Meanwhile, the GNOME team is working on creative ways to embed web applications into your desktop.

But the real innovations of the open source community are all a few layers deeper in the application stack, all the plumbing that powers the Internet. Microsoft itself borrowed its early networking stack from BSD, one of the earliest open source operating systems. Domain Name Service (DNS) and email were first implemented on the Internet using open source software (BIND and Sendmail).

The open source community tends to snicker whenever Apple claims to be innovative. Its core “innovations” were all invented somewhere else:

• The Mac Desktop interface borrowed heavily from Xerox PARC labs
• OS X uses BSD under the hood
• “Spaces” were in use in Unix systems for a decade before they arrived on the Mac
• The “Time Machine” functionality in Leopard is standard in many source code management tools

To its credit, Apple polishes these features better than anybody else, making them easier to find and use by normal people. But many, if not most of its innovations come from somebody else.

Read the previous post for more discussion about Linux support.

I started writing a response to a discussion in the latest “Linux Link Tech Show” episode, but ended up with something far too long, so I’ve split it up into 4 posts. The next post is about the TLLTS vs TWIT debate, and introduces this set of post. The previous two are about open source support–a true story of a support incident I had, and the unwritten rules of open source support. In this post, I analyze the fundamental differences between Windows, Apple, and Linux when it comes to support.

Dann and Linc had a quite spirited debate about the merits of having a company hire low-end tech support with scripts (Dann) versus having an experienced, savvy, tech professional able to really solve the user’s problem (Linc). Dann’s point was that it can easily be more cost-effective for both the support company and the end user to go straight to reinstalling a system if rebooting doesn’t solve the user’s problem, while Linc seemed to think a savvy tech person could get to the root of the issue much quicker, and brought up the point that there’s a cost to the frustration of users being put through the whole front-line support nightmare.

I’d suggest the situation is even more complex than that, but it also differs greatly between the open source projects and proprietary operating systems. First, let me make a bold statement:

Linux support is far better than Windows or Mac.

But it also has a completely different set of rules. Learn those rules, and you’ll be able to solve your problems more satisfactorily than it’s possible to do in the proprietary world. Let’s talk about these differences, looking specifically at a who, where, what, and how long.

Who can help with your problem?

Let’s take a look at who can help with your problem:

The bottom line here is that while you probably know fewer people personally who can help you with Linux, there are more options for commercial support, and you can reach people who can do more to solve your problem for free, than you can with either Mac or Windows.

The fundamental reason for this is that anybody can read the raw source code of any open source product out there, and with enough skill and talent, can solve your problem without needing to pay anybody for the right to do so. In the proprietary world, only one company can help beyond a certain point: for Windows problems, that’s Microsoft. For problems in an application, it’s the application developer.

So the next question is:

Where can you get help?
Again, let’s compare the options:

The main point of the table above is that in the proprietary world, the harder your problem is to solve, the fewer people can help you solve it. You quickly get down to one place to go, and if it’s in the operating system, it might be expensive or not possible to fix. In the open source world, it’s nearly the opposite case–it can be harder to find the simple quick answer to your question, but the harder and deeper your problem, the more places you can go to get help.

Where do you look for help first? This is the single stumbling block for most otherwise tech-savvy users new to Linux. To learn Windows or Mac you can take a class, talk to neighbors and friends, and find lots of very low-end help that way. For Linux, unless you’re friends with some hard-core geeks, you need to go online to find help. Once you’re there, it really depends on where your problem lies.

For just figuring out how to use a system, go to the forums for your distribution. The Ubuntu forums are a great place to ask general questions. Be specific about what you’re trying to do and you’re more likely to get help. Remember that this level of support is free, and people helping you are volunteering their time and knowledge. While you’re there, see if you can answer somebody else’s question–the more help you give, the more you’ll get in return.

The other fantastic place to go for help, especially for quick questions, is IRC. IRC is a system that provides chat rooms and instant messaging. Most open source projects with any community behind them have a chat room on irc.freenode.net. Install an IRC program like Chatzilla, Konversation, Pidgin, or any number of others, connect to irc.freenode.net, pick a nickname to use, and join the channel for the program or distribution you’re having trouble with. Ask your question nicely, and if you don’t get a response immediately, keep your chat program open for a few hours–not everybody is watching the channel every minute.

What kinds of problems can you solve?
Let’s look at the same types of problems as before, and look at the resolutions:

What’s the key thing in this section? Addressing actual problems is within your control, when you’re working with open source. In the proprietary world, you’re entirely at the mercy of a single software vendor. If your problem is in the interaction between two different applications, you’re really stuck — there’s nothing you can do.

But in the open source world, there’s always something you can do. You can hire anybody with the skills to solve your problem, and fix it in the software itself. You don’t need any blessing from any single software vendor.

Open formats are perhaps more valuable than open source software, for most businesses. Because this is such a compelling advantage of open source software, many proprietary programs are beginning to open up their formats to allow other software to read them–their customers are demanding it.

How long will it take to solve your problem?
I’m not going to bother with a table for this one. The answer is nearly always “too long,” regardless of the operating system.

Actually, that’s not quite always the case–it depends on whether somebody has already solved the problem or not, as well as whether the solution is a fix or a workaround.

Many, many problems in Windows are not really fixes, they’re just workarounds. The only real “fix” for a virus-infected machine is the workaround of reinstalling your operating system. The only fix for lots of other minor issues that cause your system to slow down over time is to reboot. These are not fixes, they’re workarounds.

Real fixes take a lot longer, and need acknowledgment that the problem is real. Workarounds are band aids to get you through until there’s a real fix. And there are some real differences between the entire approaches of each operating system around these fixes.

Windows is chock-full of workarounds. Because Microsoft has gone to great lengths to maintain backwards compatibility of just about everything that’s been released for Windows since Windows 95, it’s full of workarounds to keep the old behavior. Rather than fixing behavior that might really be undesirable, they’ve had to patch it with workarounds because too many existing applications turned out to depend on that bad behavior. That, fundamentally, is why Windows is so big, bloated, slow, and painful to work with.

Apple suffers a different problem: changing their closed libraries too quickly for external developers to keep up. Each release, they break lots of things, and don’t always tell 3rd party developers ahead of time. This means the number of third party developers of Apple software is shrinking–they’ve managed to alienate quite a few. So their polish and high quality comes at the price of having a healthy thriving developer community outside the walls of Apple. There’s little transparency in this process, so developers outside Apple are always playing catch-up, and having to work around these changes in behavior.

Linux has its share of problems, too. Linux does not attempt to maintain “binary” compatibility between versions, though it does its best to maintain compatibility in source code. There is endless debate about the “correct” way to fix a problem, and competition between fixes. The challenge of this is that it can be hard for application developers to keep up, especially if they want to keep their source code closed and only ship software in binary form.

But the process is completely, utterly transparent. Anybody can see the progress on any fixes to any part of the system, and can jump in with their own solution at any time. It’s a true meritocracy, with those doing the actual work and providing the best solutions winning out over time.

The whole open source ecosystem is nimble enough to provide real fixes to technical problems, rather than just simple workarounds. If the source of the problem is design, it can take a long time to get the right design in place and resolve all the issues that changing the design causes in other applications. But when a bug gets fixed, it’s really fixed and usually doesn’t appear again.

In my early Linux system administration days, when I was first trying to set up a mail server with spam filtering, I ran across a really puzzling bug in Dspam, the software I was trying to get working. While all the other users of the software were getting great results, with Dspam catching 99%+ of all their spam, it was only catching about 70% of my spam after quite a bit of training.

I posted my results, and confusion, to the Dspam mailing list. The original developer of this software (which has thousands of users), Jonathan Zdziarski, responded that that did not sound right. He asked if he could log into my server and see what was wrong.

I created a test account for him, and logged in to the same screen so I could watch what he was doing. As I watched, he put debugging messages into his code, ran several tests, and within 10 minutes had identified the problem: the 19-digit number he put into the database (MySQL) was different than the 19-digit number he got out. It was a storage error in somebody else’s software causing the problem, a rounding error. He filed a bug in MySQL, and it was fixed within a month or two. But he also had a workaround for me: change the data type to a set of characters instead of an integer. Problem solved.

What’s extraordinary about the open source community is that this level of support happens all the time, every day, without charge, in hundreds, thousands of projects out there. People that can get to the bottom of a problem and fix it at the source, not just provide a workaround, are directly reachable and motivated to see their software work as well as possible. They’re not hidden away from the public behind a large corporation, unreachable with layers of clueless support script readers stuffed between you and them. Here are some rules for getting open source support directly from the projects:

• Before asking anybody, do your homework. Use Google, read the project FAQ, make some attempt to learn the basics without pestering people with questions they’ve already answered hundreds of times. Nine times out of ten, your problem has already been encountered and somebody has figured out a workaround.
• Limit the scope of your question to the fundamental problem. Get to the point. I’m obviously guilty of being tremendously long-winded at times, but unless your question is right at the top and asked directly, you’ll probably get ignored. Developers are busy, they don’t want to read a novel, but they’re usually happy to answer a question.
• Provide supporting details after asking your question. Many programs will create a log with lots of information that can help somebody diagnose your problem. Find what looks relevant in the logs. Specify what version of operating system, distribution, application, etc. Specify what you were trying to do, what you expected, and what really happened. But provide this stuff after asking your initial question–people aren’t going to wade through a long email to find your question.
• Contribute something. The easiest way you can contribute is by answering other people’s questions. The whole thing works because people help each other, and this help goes both ways. If you always ask questions and never answer anybody else’s, or provide any other sort of contribution, you’ll eventually start getting ignored.
• Always, always, be positive, respectful, and polite when asking your questions. Developers have a lot invested in their project, and insulting it won’t gain you any favors. Developers are under no obligation to help you either–you haven’t paid for it. Common courtesy is valuable. Complements are welcome.
• Be patient. Sometimes the person who has the answer to your question is away from the computer. Usually you’ll get your problem solved quicker than you would calling some tech support line, but there are times it’s going to take a while. If you don’t get any response in a reasonable period of time (judged by how active the list or forum is, reasonable could be a couple hours or a couple days), there are several likely reasons: You haven’t been specific enough in your question; you’re in the wrong forum (e.g. users when it’s a developer question); nobody else is trying to do what you’re doing (in which case you may need to hire someone with the right skills); the project is dead (it happens sometimes–find another one); or the developers are swamped (give them more time, or come up with a new scenario that sheds light on your problem in a different way).

That list describes how we get open source support at Freelock. Aside from a couple unsupported hardware devices, or issues with proprietary programs, we have yet to get stumped, in over 6 years of extremely heavy Linux and open source use. We’ve never paid a dime for this support, though we have provided help to many others in return.

At Freelock, we’re always trying to figure out ways to do things better. Recently I started digging into a developer tool that’s making, as Bryan over at the Linux Action Show would say, my head explode.

For a long time, we’ve managed our custom code projects and business documents in a central repository, called Subversion (also known as svn). Subversion is relatively easy to understand–it’s like having a library of files you can check a copy out of, do some work on it, and then check it back in. Subversion is the librarian that tracks who has copies of what, and when they checked it out. So if Erik checks in changes to a brochure, and then Jill goes to submit changes to the same document, Subversion will say “hey wait a minute, that document has already been changed–you need to make sure you put Erik’s changes in your document before I’ll let you put in your document.”

This is great for managing conflicts between people working on a single team, or for code that is being developed in relative isolation from the rest of the world.

The problem is, we’re doing more than that–we’re taking code from various open source projects and either customizing it or building new applications on top of it. And so when the outside projects get updated, we need to manually update anything we’ve written that depends on that code. There is no longer a single repository where we control our code–there is our code library, plus another one for every project we use.

This makes managing add-ons for projects like Joomla or ZenCart quite challenging, because our add-ons get scattered throughout the filesystem to be able to hook into the right place. And if we have to touch a core file, we’re going to end up needing to re-implement our change with any update to that core file.

There are other issues we run into, managing our code and hosting, all of which take fairly time-consuming, manual intervention. Here’s the list:

• Since we host and provide security updates for Joomla, Word Press, Zen Cart, Drupal, and others, we need to upgrade dozens of installations any time there’s a new release that has a fix for a security vulnerability. With Joomla this has happened quite a lot, and every Joomla installation needs to be upgraded individually–and tested. And since each installation is slightly different, we can’t manage them easily within a single repository, while updating the underlying code.
• Templates, modules, components, blocks, themes, plugins, and whatever. Developing these types of add-ons are our bread-and-butter. But code for these often get scattered across an installation, making it quite difficult to manage just our add-ons while we develop them, or roll back to earlier versions if there’s a problem.
• The Dojo Toolkit, and builds. We’re doing a lot of development with Dojo right now, to add desktop-like functionality such as trees, sortable tables, right-click menus, animations, and lots of other really cool things. However, if you don’t “build” the code after you write it, it’s painfully slow in a web browser. And due to the nature of how Subversion works, you can’t easily store a built Dojo tree if you ever want to change it again. Which means you’d need to build it every place you deploy it. And on some computers, it can take a long time to build–on our demo server, one of our projects currently takes 8 minutes.
• As we get more directly involved with open source projects like LedgerSMB, we’re finding the need to change core files while we hack away at some particular feature. To do this, you create a branch of the code, work on your feature, and then merge your changes back into the “trunk.” If you don’t have access to save directly to the project repository, doing this gets a lot more complicated.

Git to the rescue. Git solves all of these issues. Read on for a technical discussion of how.
Read the rest of this entry »

Unbelievable. Microsoft was one of the first places to support WebDAV, and after a little investigation, looks like they’ve completely changed how they support it–with security implications, and an amazing amount of brokenness…

At Freelock, we’ve used WebDAV to allow our clients to access to our servers since day 1. FTP is fundamentally unsecure, and as business level hosts, we refuse to allow that. SFTP is a really good option, but it does require us to set up local user accounts on the server and allow a higher level of access–something we would prefer not to do on our shared servers. WebDAV has long been the clear answer to this, supported by every major operating system with no extra add-ons, and also supported by most web development tools natively. That is, until last year, when Microsoft completely changed the way they do WebDAV. It even breaks compatibility with their own Sharepoint software!

Without testing this fully, this appears to be the situation, version by version:

* Windows 98, 2000, XP (does this still work in SP2/3? Dunno):
- Use Internet Explorer. Go to the File -> Open dialog, check the box to open as web folder, enter the WebDAV URL, and open.
This works really well, though if you bookmark it, it will open it as a web page, no longer a web folder.

* Windows XP SP2:
1. Apply a registry hack to enable basic authentication
2. Open Windows Explorer, and go to Tools -> Map Network Drive
3. Enter the path to the drive (you cannot use https unless you have Office installed) and a drive letter to map. Alternatively, use NET USE on the command line.

* Vista:
1. Apply registry hack to enable basic authentication (or set up server to use Digest authentication, and strip domain name out of user credentials)
2. Set up server to not reject requests to anywhere in the path for OPTIONS and PROPFIND requests

Read on for more details.
Read the rest of this entry »

Packtpub is running a sample from a developer’s guide for customizing SugarCRM. The author describes how to set up hooks for particular modules to build a custom workflow.

Custom workflows are a feature that is limited to the proprietary version of SugarCRM–they have not been available in the open source version. With custom development using techniques illustrated here, you can add your own workflows.

This looks to me like it’s written specifically for versions of SugarCRM before version 5. I haven’t had a chance to find out whether the same basic techniques would apply–SugarCRM 5 changes a lot of things from earlier versions, primarily with email handling and storing customizations in the database rather than scattered around files. The basic approach should work, however…

Developing a Simple Workflow within SugarCRM

Some handy tips for users of OpenOffice.org, looking to get away from the mouse…

Ten fantastic keyboard shortcuts in OpenOffice.org

1. You want to make sure nobody will be able to read your documents in 10 years
2. You want to help your buddy who works for Microsoft have enough income to buy a private island in the Carribean, because maybe he would invite you to come for a weekend
3. You feel sorry for the PC on the Mac commercials
4. Your buddy is buying it for you from the Microsoft company store, so you’re actually saving hundreds of dollars! You can’t get those types of deals on free software.
5. You hope that the extra emails it takes between you and your customers, partners, and vendors to get formats that they can open will improve your relationship with them
6. Having the newest software from Microsoft makes you cool
7. You want to extend Microsoft’s monopoly on the desktop, it’s just easier that way
8. You have a big technology budget, and can’t think of any better way to spend those dollars
9. You already spent the money on it, may as well force others to pay their Microsoft tax, too
10. You’re a big fan of Survivor, and like being dropped into an unfamiliar environment and having to figure out all over again how to do the things you need to survive
11. (Bonus!) You don’t know a better solution exists

If your reason for purchasing Office 2007 is #11, drop us an email and ask us how open source software can make your business run better.

I’ve been struggling to get Project Auriga to set HTTP Auth from a nice pretty login form, and think I have it working.

What follows is a very technical discussion–if you’re a business reader, you should probably skip this post…

HTTP Auth is a specific mechanism for handling authentication. HTTP Auth is built into Apache and IIS, and so the server can handle authentication purely through configuration, offering many different back ends for storing the data. Browsers also handle HTTP Auth natively, popping up a normal login box whenever it gets a Basic Authentication request from the server. But this login box is ugly, and doesn’t provide a friendly experience to allow people to create an account, get a password resent, or anything–it falls back to a basic error page. You can, of course, customize the error page, but not necessarily help people with the password login itself.

There are several benefits to using HTTP Auth, though. First of all, other applications on the same server can accept the same credentials, allowing you to sign in once and access multiple applications without having to log into each one. Secondly, you can set up stronger authentication methods, such as client-side certificates. Also, you can configure the server to protect large parts of a web site very easily, reducing exposure to information disclosure.

So how do you make a sign-in form on a web application set http auth? Browsers do not allow you to access these settings via script. You can use an XmlHttpRequest object to set authentication, but only after the proper challenge has been sent from the server. The biggest problem is, if the server sends this challenge twice in a row, your browser will intercept the second request and pop up the ugly password prompt. So designing a form that keeps this login prompt from popping up under most circumstances is quite the challenge.

The gist of the issue is that while you can open an XmlHttpRequest object with a user and password for http authentication, the browser will only actually use those credentials after the server has rejected a request. The process looks like this:

1. Your script creates and sends an XmlHttpRequest with http auth username and password.
2. The browser submits the request to the server, without sending the username and password.
3. The server responds with 401 requires authentication, and a WWW-Authenticate header specifying a realm.
4. The browser looks in its cache to see if it already has http auth set for that domain and realm. If it does, it sends those credentials, NOT THE ONES you specified in your XmlHttpRequest. If it does not have those credentials, only then will it set http auth to what your script asked for.
5. The server responds. Generally, if the username or password are incorrect, the server will repeat the 401 response, and WWW-Authenticate.
6. The browser gets its second 401 in a row, and pops up its password box. Your script never gets a chance to intercept this. So if the stored http auth credentials are wrong, or the user mistypes the password, their browser takes over and you get a password prompt.

How do you handle this situation? It turns out you need to engage in some trickery on both the client and the server.

Here’s a basic flow of how you need to handle this, from both the server and the client perspective:

1. First, collect the credentials from the user, and create your request as outlined above.
2. Browser sends request without credentials.
3. Server responds with 401 and WWW-Authenticate.
4. Browser sends cached credentials, if they exist, or your credentials if not.
5. If credentials are accepted, server allows log in and responds with 200. If credentials are not accepted, server returns an error code OTHER THAN 401, and does not send a WWW-Authenticate:
   1. We use 403 not authorized for a credential failure here. You might also use 400 Bad Request.
   2. Because the response was something other than 401, your browser caches the bad credentials.
   3. XmlHttpRequest status reflects the error condition.
   4. Your script checks the result for the error your server has returned. Now comes the crucial part:
   5. Your script submits a new request with different credentials to some server location that will return successfully. For example, we call a login method on our application, passing username “public” and password “?”.
   6. The browser sends the new credentials and submits the request.
   7. The server returns 200.
   8. The browser updates its http auth cached credentials with the new bogus ones.
6. Now you can present an error to the user, and ask for new credentials.

The key to the above process is that if the browser gets two 401 responses without having a 200 somewhere between, it will pop up its password box and there’s nothing you can do about it. So the key is to use a different error code to indicate bad credentials, and do an intervening request that will return 200 so that you can re-authenticate.

Logging Out
You cannot really log out of HTTP Auth. But you can change the credentials to a known bad user. That’s a key technique we use to effectively log out of an application, and we re-use this method to reset after bad credentials.

On the server
I’m very much still in development with this. You can see the server side code for Project Auriga logins here.

In this system, we do set a cookie after successful login, to keep from having to check credentials again. This script also allows for cookie-only logins without using http auth. The important bits:

• action=logout: if this is called, the script always returns successfully. This allows the client script to provide new bogus credentials. It passes a username of “public” to log out completely.
• action=httpauth: if this is called, and there are no http auth credentials or the http auth username is “public”, return a 401 and WWW-Authenticate. This is always the first request from a browser, and triggers the browser to re-request with the credentials.
• action=httpauth, with http auth username set, and it’s not “public”: The second or later requests, we never want to return a 401 or the browser will pop up its password prompt. So we return 403 (or 400) if the credentials are bad, or allow the script to continue processing if its good. In this case, our authenticate method returns true if credentials are good, false if the user is not found, and throws an exception if the credentials are bad.

That’s basically what you need to do on the server side. Now for the client.

Client-side logins
We’re using the Dojo Toolkit extensively in Project Auriga, so the login functions are using dojo.xhr* requests to wrap the XmlHttpRequest objects and provide convenient callback functions. You can see our login code here. Key items:

• auriga.login is called by the login form. Note that if this is the first time to this page, the dojo.xhrPost actually happens twice: first time with no credentials, and the second time with them. If the second post is accepted, auriga.login_complete is called. If the second post returns any kind of error, auriga.login_err is called.
• auriga.login_complete is easy… it just redirects to wherever the server response designates.
• auriga.login_err is the real trick here. If it detects the error code we’ve chosen for bad passwords, it immediately calls the server logout method, to get a good response so the next time the browser gets a 401, it won’t immediately pop up the password box.

You can see the code in action on our demo server.

Other notes

• Actually doing single sign-on is hard. We’re trying out different strategies for detecting whether a user already has http auth set, by calling our login method once on page load, but haven’t gotten that figured out. In our current script, just clicking Login with the form blank but authenticated elsewhere on the same domain and Realm, will log you in with your existing credentials.
• Because your browser stores credentials based on the domain and the realm together, all applications that you set to share these items must accept the same credentials. If you have a different password on a different system on the same server, you must set a different realm, or logging into one will log you out of the other.
• If you want to require http auth, but not Javascript, I suggest submitting something different to the server using Javascript to identify this type of request. Perhaps show your form only when Javascript is available, and when it’s not, have a link to a protected page to let your browser go ahead and show the password dialog.
• Using http auth can actually allow users to disable cookies, if your application is RESTful. In Project Auriga, the session login script supports either–the client pages and logins work with either a cookie or http auth. The login process attempts to set http auth and a session cookie. On subsequent attempts, it uses the cookie to avoid re-authenticating every request.
• Finally, a note on security: Basic Authentication provides no protection against passwords being sniffed over the network. If you need a secure login, be sure the server conversation uses SSL–otherwise neighbors on your wireless network can easily sniff out your password. HTTP Auth does not make your application more secure–it just makes it easier to share authentication with other resources on the same server.

LugRadio has a very interesting discussion in their current podcast about the role of a community manager, in creating a vibrant community around an open source project. They came to the conclusion that each project needs a leader that people trust to take the project in the right direction, someone to be a diplomat to resolve issues among people in the community and keep everyone rowing in the same direction, and a strong technical lead to solve the hard problems.

This sounds quite similar to the challenges a small business faces. “The E-Myth Revisited,” by Michael Gerber, is essential reading for anyone wanting to start a small business, and some of the same ideas apply to building a successful open source project.

Basically, Gerber says you need to have 3 personalities in your business:

• The entrepreneur, the person with the vision to drive the business/project forward, always a step ahead working on what’s next
• The manager, somebody to make sure all the work gets done on schedule, delivered on time, and that everybody is working in the same direction with the same priorities
• The technician, who actually does the work

And small businesses, like open source projects, are almost always started by technicians, people who just know they can do a better job than anybody else, so they set out to do it themselves. The reason why so many businesses fail is because the founders spend all their time building something without setting enough overall direction or managing their cash flow. Sound familiar?

The key to creating both a successful business and an open source project is balancing out these 3 personalities and making sure all are represented to an appropriate degree.

Patrick asks,

Why not OpenSuSE, instead of Ubuntu?

At Freelock, we provide a maintenance service contract to manage Linux servers. For a fixed monthly fee, we provide monitoring, system updates, application updates, and our help recovering anything that goes wrong with an upgrade. We’re looking at adding disaster recovery to the mix, raising the price to cover the cost of backing up all of the data and providing varying service level agreements on how soon we will recover your machine from a total loss. But for our base price, we only support Ubuntu and CentOS, with a preference for Ubuntu. So Patrick asks, why not OpenSuSE? Here’s my reply:

Hi, Patrick,

There are literally hundreds of different distributions of Linux, and 7 or 8 that are very widely deployed in server environments. The reasons Ubuntu is our preferred distribution include:

• No separate version of operating system–Red Hat and Novell/SuSE keep some of their stuff for their commercial version
• Commercial company backing it (Canonical, LTD), support contracts available
• Strong community support, lots of friendly help available
• “Long Term Support” releases, with a commitment by Canonical to maintain security releases for 5 years on designated versions
• Superior upgrade path when a version reaches end-of-life
• Nice balance between cutting-edge versions of new software, while making sure they’re stable
• Easy package management, well-built packages

OpenSuse has no commercial support available, and no commitment on the part of Novell to provide long term support to their free versions–you have to buy SuSE Enterprise Linux to get that level of stability/support, and that’s not the same software–they bundle older versions in their enterprise distributions that often don’t have features we’re coding on top of…

The upgrade path is another nice feature–we’ve had very good success upgrading Ubuntu boxes in place, without having to install an entirely new system and migrate the data over.

We support CentOS as well, which is a free version of Red Hat Enterprise Linux, because some of the software our clients run are only available for Red Hat, so we have to… CentOS doesn’t have as good an upgrade path as Ubuntu, so we don’t include system upgrades at software end-of-life with that. We have supported SuSE boxes in the past, but I think we’re down to a single one running some legacy software. The main reason we discourage it is to streamline our processes–it’s much easier to administer a bunch of the same operating system, than having every box be a one-off system.

We’ve been doing this long enough to have to upgrade several servers because the OS has reached end-of-life and there are no more security releases available. Of the free distributions, the only ones we can deploy with confidence knowing we won’t have to upgrade for at least 3 or 4 years are Ubuntu, CentOS, and Debian. While there are many other solid distribution choices you could make, none of the others quite stack up to meet our needs as well as Ubuntu and CentOS.

This is a new feature we’re starting: Ask Freelock! Have a question about using open source in business? Drop us a line, ask us a question. We’ll do our best to answer.

It’s been a while since I posted here, but it’s not for lack of things to write about. There’s plenty going on, and I have several blog posts/rants/articles pretty much ready to post. Just haven’t had time to get them up.

We’ve got a bunch of irons in the fire, and have much of the hard work done. That means I’m going to be writing more, letting the world know about some of the cool stuff we’re doing over at Freelock Computing. Top of the list is Project Auriga, which is quickly becoming a really cool time-tracking, billing, project management application that highlights the Dojo Toolkit to do all kinds of fun stuff right in the browser.

After a year of not getting why I should, I’ve started Twittering as an experiment in transparency and an exercise in posting more regularly.

We’re working on a major overhaul of the main Freelock.com web site, moving it from Joomla to Drupal. We may also move this site to Drupal, and turn it into more of a directory of open source applications, along with discussions about the various solutions so that people can find straightforward information about the state of open source business software.

Finally, I do plan to blog on a more regular basis going forward. And I’d like to make this a useful resource for people looking for answers. So if you have a question about using open source in small business, please post a comment or email me at john at freelock.com, and I’ll do my best to answer them…

A couple weeks ago, the EU slapped Microsoft with a $1.35B fine, less than a week after Microsoft had made a big fanfare about their new “open” policies.

Todd over at Napera asks,

Certainly the terms Microsoft has been offering companies since the EU decision in October 2007 are extremely reasonable. Given Microsoft’s new open protocol documentation and their patent pledge for open source developers, what’s not to like?

I haven’t seen any pundits or commentators in the US defending the EU decision. If they are, what are their substantive points?
Napera Networks » The EU crashes Microsoft’s party

Hmm. Let me see if I can shed some light…

First of all, Microsoft wants you to believe that open source developers are all a bunch of hobbyists creating code for no pay. Their recent pledge is to not sue open source developers developing non-commercial systems.

Boy, what a deal that is… Microsoft gets all this open source development for free, right? And they can still charge anybody who dares to compete with them commercially? Great deal–for Microsoft.

The reality is, a great number of open source developers engage in commercial activity. Many successful open source projects are backed by commercial companies. OpenOffice.org, JBoss, SugarCRM, MySQL, PHP, the list goes on. By definition, open source means you can use and redistribute software for any reason, including commercial activity–by excluding commercial entities from patent protection, Microsoft is making a marketing play without making any real concession.

Their real agenda here is to give the illusion of openness so they can get their OOXML format approved as an open standard, without giving up any control over the format. Never mind that they were part of the committee developing the ODF standard and didn’t bother to contribute. Microsoft has no interest in working with other companies to develop an open standard–they want to be proclaimed the standard so they can demand “reasonable royalties” from everyone else, and all the rest of this is just marketing spin to try to put one past the EU, who so far have managed to see through all this bull.

Secondly, Microsoft seems to think the only healthy software marketplace is one that revolves around it. It got where it is by using all sorts of unethical, hardball business practices to drive its competition out of business. This is classic anti-competitive behavior, and is the reason it’s been the target of so many actions.

Microsoft has built a large ecosystem of developers and service providers all gathered around the MS stack. Yet you look at the stack and there’s not all that much vibrant activity, at least not compared to the open source bazaar. Take content management systems as an example. In the Microsoft world, you’ve got Sharepoint. You’ve got a couple of ASP.net open source projects like Dot Net Nuke. And that’s about it.

On the open source side, you’ve got overwhelming choice, and some of them quite great. Instead of one size fits all, you can go shopping for just the features you need, and get it all for a great price–free, if you have the technical ability to make them work. Joomla, Drupal, Plone, MediaWiki, Typo3, Postnuke, Word Press, Serendipity, several hundred different options for you to choose from. Which looks like a more vibrant marketplace, with more competition and more choice?

Ah, but then you ask about standards–isn’t it great that Microsoft makes it so simple for you to do what you need to do? You don’t have to make all those choices. Soviet bakeries were so much better than American supermarkets, too, huh?

Oh, but wait. Who invented http, rss, the world wide web, email, the spreadsheet, the database, the word processor, or anything else we use our computers for? Hint: not Microsoft. And the first few things in that list were created in universities and other open source, collaborative environments.

Finally, to speak directly to the EU’s decision, how does letting Microsoft get away with their monopolistic, anti-competitive behavior help the EU in any way? By protecting their software industry from Microsoft, they’re fostering an environment that can lead to a much more competitive, fertile ground to grow their own software industry.

No other industry (other than possibly the utility companies and the railroads) have had a single company that so dominates the industry like Microsoft dominates software. How can that be a good thing? I know from personal experience a half a dozen companies that Microsoft crushed, and for the most part their technology with them. This mono-culture of software has led to all the trouble with spyware and botnets and through them, spam. I don’t think we’re better off with our Redmond overlords.

I for one was happy to hear about the EU’s decision, glad they were able to stand up against monopolistic practices and do something to actually help their software industry thrive.

PHP development is one of our specialties at Freelock Computing. I’ve written quite a few PHP applications, some from scratch, some starting with other people’s code, some as extensions for open source projects. I’ve also read a lot of criticism of PHP, and while some of it comes from knowledgeable programmers expert at PHP, most of it is uninformed hogwash. So in this post, I’m going to dispel many of the myths about PHP code, and identify its real strengths and weaknesses. Most myths have a kernel of truth in them somewhere, so I’ll try to set the record straight by identifying why PHP has each myth. Ready? Let’s get started.

1. PHP is crappy because there are so many crappy PHP programs.
This seems to be the biggest reason people think PHP is a bad language–there are a lot of bad PHP programs out there. Why is this? Probably because PHP is so accessible and ubiquitous that a lot of people without a programming background use it to learn programming. I’ve worked with programmers inside software companies who have much more formal background, or at least experience programming with others on a team. With somebody to guide them, they quickly learn the pitfalls to avoid, best coding practices, and development methodologies.

Most PHP coders on the other hand started out as web designers, putting together a web page for their neighbor, or their family, or a club of some kind. They have no formal training, no experience working on a development team, no guidance or knowledge about what makes for quality code. The result is inevitably spaghetti code, chunks cut and paste into place without real understanding of how they work, people fiddling with lines until it gives them the result they’re looking for.

Naturally, this leads to a lot of crappy software out there, riddled with security holes, maintenance nightmares, poor performance, and many other problems.

That does not mean the language itself is at fault. There are plenty of well-written programs out there that do an excellent job of doing the task they’re designed to do.

Result: Busted. Bad programmers does not mean its a bad language.

Let’s get a bit more specific about these code quality issues.

2. PHP is crappy because it’s hard to read all that HTML mixed in with programming logic.
Some argue that PHP is this way because it is a template language–it was designed to be an easy way to add basic programming functionality to a web site. And while that was its heritage, PHP has grown into a full-fledged powerful language capable of most anything you’d do with any other language.

Nothing in the language dictates that presentation code (HTML, Javascript) needs to intermingle with business logic. I consider the best programs to have a clear division of responsibility between these areas. We use a strong Model-View-Controller (MVC) architecture when creating custom applications, the same architecture provided by many frameworks, and advocated by experts for many other languages. And we’re hardly alone in this.

We use the Smarty template system to separate out the templates into a presentation layer. Our model is usually made up of fairly lightweight data objects that own the corresponding database tables. The controller layer is typically a dispatcher of procedural code, often with helper controller objects. You can apply most design patterns to PHP as readily as other object-oriented languages.

Now, the tools you use to develop PHP don’t enforce any of this. Unless you’re using a framework, you need to create all this structure yourself. But we don’t like HTML mixed in with our business logic ourselves, so we don’t do it.

Result: Busted.

3. PHP is crappy because it’s easy to hijack with all those global variables.
Funny how people try to create all these really easy ways to do things that turn out to be large mistakes from a security point of view. Microsoft has done this over and over. PHP has two particularly annoying “features” that have turned out to be security nightmares, originally there to make it simple to program: register_globals, and magic_quotes_gpc.

register_globals is a setting that takes any parameters passed in a request and automatically turns them into a global variable you can use in your script. The problem with this is that it’s very easy for an attacker to pre-define a variable that the script assumes to be unset. As I was learning to program in PHP, I wrote a 500-line script to check and double-check that each variable I was expecting from the browser was legitimate, and that all of my other variables were suitably protected.

At its worst, register_globals turned out to allow an attacker to include a malicious PHP file from a remote web server before your script even started, by setting an autoload variable for a particular module.

register_globals is evil.

But its vulnerabilities are widely known, and PHP has been set with register_globals turned off for several years now. It’s going away entirely in PHP 6.

magic_quotes_gpc is more of a pain. It was added to help prevent SQL injection attacks, and what it does is escape all of the values you receive from GET, POST, and COOKIE parameters, adding backslashes in front of any backslash or quote to make it so programmers who pass these variables straight into a database query have some protection built into the language. But it causes a lot of extra work, because your script doesn’t know whether this is on or off. If it’s on and you escape your strings, you end up with extra slashes in front of everything–and you end up with backslashes scattered all over your pages. We end up checking the setting of magic_quotes_gpc, and if it’s on, stripping the slashes before the rest of our script interacts with it.

For any experienced PHP programmer, these are solved problems.

Result: Busted, but there is valid criticism here.

4. PHP code doesn’t scale well.
Nonsense. This is purely myth. Here are some of the most popular sites on the Internet that run on PHP: Facebook, Flickr, Wikipedia, Digg, parts of Yahoo. All of those are in the top 20 most visited sites on the Internet.

Result: Busted. Very busted.

5. PHP is mainly a vehicle for Zend to get business.
I didn’t hear this one until just recently. Zend is a company with a strong stake in PHP. It controls a lot of the code, it has a decent editor with a debugger, a powerful framework, and a PHP accelerator available as proprietary add-ons. I’ve had a couple developers suggest that Zend has such a controlling interest in the language that it keeps others out, and you have to buy from Zend to make PHP work best.

Yet this ignores the other options out there. Zend does not have a monopoly in any of these areas. There are several other editors with good PHP debugging support, dozens of frameworks, and a handful of PHP accelerators out there, several of them completely free and open source. Now if you’re trying to change the core PHP language, you may need to work with Zend, and I have heard they aren’t necessarily the easiest to work with–they don’t readily accept changes to core features, and a few developers have left the PHP project because of disagreements over the direction of PHP. And some of these have been serious, related to hardening PHP to prevent some of the preventable security attacks through the language itself.

But as a PHP user, these issues seem far removed. PHP 6 is in development now, promises some decent improvements such as Unicode support and removal of some of the vulnerable settings.

Result: plausible, but not relevant to most PHP developers

6. You can’t compile PHP, so it will always be slow.
PHP is an interpreted language, and it doesn’t have a built-in compiler. The same is true of other web languages, at least Perl. Python has a built-in runtime compiling system, so you get compiled byte-code without having to do anything. I don’t know that much about Ruby in this area.

But you can accelerate PHP quite similar to Python, by adding an accelerator. Zend has a proprietary one. We use eAcclerator on our servers, and there are several others out there. These provide what is called an “opcode cache.” When PHP is executed, the interpreter makes two passes: first a conversion to native bytecode, and then execution of the bytecode. An opcode cache stores that first pass to disk, so subsequent calls can use what is essentially the same as compiled code. While it’s not permanent, and probably not as efficient as other compiled languages, it does seem to allow our servers to accommodate about 40% more traffic before bogging down.

Combining this with other caching strategies can allow PHP sites to scale up to serve the largest sites.

Result: Plausible, but workarounds available.

7. You can’t develop in PHP as fast as other languages. Like Ruby on Rails.
Ok. Now we’re getting to the ridiculous one. First off, Rails isn’t a language, it’s a framework. And by many accounts, it’s a good one, providing a lot of really powerful features right out of the box. It might have set a new high standard for developer-friendly frameworks. But it’s hardly the only one out there, and because it’s open source, many of the conventions it established have spread widely to other frameworks as well. CakePHP is a framework that aims to be the Rails for PHP.

Rails has its downsides as well. The CEO of Dreamhost has an interesting post about his experiences trying to get Rails to scale. While it may be fast to develop in, it may be at the expense of running fast enough to handle large loads. You also need to learn Ruby, which has quite a bit different syntax than PHP. PHP is quite similar to C, Java, Perl, and other popular languages, so it’s immediately familiar to many other programmers.

The biggest problem I have with Rails is the dogmatic nature of many of its practitioners. And it has gotten such widespread buzz in such a short period of time that in some ways it’s become the new PHP, the new pet technology by a lot of inexperienced programmers due to a low barrier to entry. If you’re a web designer and not already a programmer, you would probably choose Rails to get started in today, instead of PHP, because of all the hype. I think that’s going to lead to the same proliferation of lousy code that permeates the PHP landscape now.

While Ruby may be a nice language, there’s a lot more support for PHP right now, in available talent, web servers, scaling experience, and breadth of libraries available. And by starting with an application that meets 90% of your needs today, you can work on what makes your particular problem unique. Since so many applications and libraries are available for PHP that need very little customization to meet many business problems, developing from scratch with a powerful framework isn’t necessarily the fastest way to get the job done.

Result: Busted. While Ruby on Rails is nice, it’s not the only way to build an application quickly.

9. PHP is only good for web applications–it’s no good for anything else.
PHP was built to be a web application language, but it has a command line interface, a GUI toolkit based on GTK, and other features that mean you can feasibly write just about any kind of application you can think of in PHP.

However, nobody does. I have not seen a single PHP desktop application in use. While we do use it for scripting a few web server-related tasks, they tend to be maintenance tasks or a forked process from a web application. There aren’t lightweight PHP libraries optimized to run on embedded devices.

As I look over options to write software for the OpenMoko platform, for example, PHP does not appear to be a compelling option. Likewise, it seriously lacks the ability to interact with hardware or much on the OS level without calling a shell to start some other program. Perl has long been used for these environment, but Python has been taking these environments by storm.

So while it’s possible to use PHP for purposes other than web applications, it’s not convenient, conventional, easy, or widely done.

Result: Confirmed.

10. It’s not a real language–you can’t do proper object-oriented designs, objects are copied, etc.
PHP was never designed by computer scientists. You could argue it wasn’t designed at all. It was built from the beginning to solve a specific problem: to make active web sites. And it’s successful because it’s done that exceedingly well.

Over time, it has accumulated modules that do just about anything you might want to do in a web application, from talking to just about any database system out there to requesting pages from other servers to processing financial transactions to generating images and even PDF files. It added object orientation in PHP 4, and made it much more robust and similar to other languages with PHP 5. While it still doesn’t do multiple inheritance, or threading, or similar advanced programming techniques, you can implement most common design patterns, objects are now passed by reference, there are constructors and destructors and all sorts of things that give it as much power as any other language for most web applications.

While PHP certainly has its shortcomings, for the vast majority of web applications, it provides exactly the right combination of sufficient power to do the job, and a relatively straightforward way of getting the job done.

Result: Busted, for all but the most specialized applications.

That’s enough for now. In a future post, I’ll discuss the major drawbacks and benefits of PHP. Stay tuned!

There are thousands of languages out there, but only a couple handfuls are used for web applications. Of these, PHP is a runaway success. Yet I constantly see it criticized by developers of other languages, often for completely untrue reasons. PHP has a bad rap, and while it certainly has its pitfalls, there’s many good reasons it has become such a popular language for web applications.

I consider there to be three major sets of languages currently used for web development. When talking with developers, you’ll usually find them gravitating to one of these three spheres: the Windows world of Microsoft ASP, ASP.NET, Cold Fusion, C#; the Java world; and the LAMP world. While some programmers cross between these, you’ll usually find people that are best in one particular area.

The Microsoft world grew out of ASP and Cold Fusion into the current .NET technologies. There is now an open source version of .NET called Mono, backed by Novell, which makes these technologies cross-platform. They’re mainly used by Microsoft and its partners, and small proprietary software companies in all sorts of vertical industries. Very few .NET applications are open source, compared to the other technologies.

The Java world seems to dominate the large enterprises. Companies that work with IBM extensively end up with Java-based enterprise applications–and there are a lot of them. Java was the “next big thing” in the second half of the 1990s, but it only seemed to gain a real foothold in large business. Quite a few of these applications are open source, and there’s a lot of applications large and small you can download freely and deploy–or pay thousands of dollars to a middleware vendor to have them get you running. Java has a wide mix of open source and proprietary applications available.

LAMP stands for “Linux, Apache, MySQL, and PHP,” though there are other P’s out there like Perl and Python. This describes the other major technology stack used in the web world, and follows the Unix design of small pieces loosely joined–you can substitute MySQL with Postgresql, Apache for another web server, and many other languages for PHP. There are far more open source applications available on the LAMP stack than the other two combined, mainly because the barrier of entry is really low–all you need is a spare old computer to install the stack, and all the software is free.

There used to be another popular language, TCL, running on the AOLServer, but you really don’t see much in that these days.

If you’re developing a web application, you can use any of these technology platforms to get the job done–in a web environment, they are all pretty much equivalent. Java and .NET have better support for desktop applications, but if your main interface is a web browser, there’s nothing you can’t do in LAMP that you can in the others.

LAMP is a family of technologies, with more variety than the other stacks. For the language, besides the ‘P’ languages of PHP, Perl, and Python, there’s also Ruby that has gained a lot of popularity lately. MySQL and Postgresql regularly vie for the database slot. Apache pretty much has the web server part locked up, but Linux can even be replaced with Windows to make it the WAMP stack and you can still run most of the same programs.

So why group technologies into these stacks? Mainly because they work well together on the same system. This boils down to the web server part of the system. If you’re using Microsoft IIS for your web site, you’ve got .NET, and while it’s possible to add PHP or Perl, it’s not commonly done. For Java, you need an application server. But Apache makes it pretty easy to plug in all sorts of the open source languages as modules, and run a bunch of them simultaneously. Much of these differences are due to historical and cultural differences, not really technical. It’s just that these particular sets of technology are regularly used with each other, so they’re going to be easier to get running and working correctly.

Let’s take a closer look at the LAMP family. Like many families, there’s in-fighting and bickering over who is best at what job. Postresql people look down their noses at MySQL, which they clearly consider to be inferior in just about every way (with some justification). Perl people wonder why others program in anything else, Python people think the other languages make programming too difficult, and Ruby programmers pride themselves in writing the shortest code to get the problem solved. They all sneer at PHP, regarding it as a toy language not capable of real programming. Yet you’ll find more open source web applications written in PHP than all of the rest of them. Why is this?

Read my next post to find out why.