When I heard it was up to me to pick out a bug tracker for the team, I was a little nervous, but not surprised. I joined Proletariat knowing that issues were being tracked in Google Docs and that this would have to change quickly. When I heard I’d then have to write a blog post about it, I locked up and started crying. Here it is, weeks late:
My first instinct was to go with Atlassian’s JIRA, a tracker most of us had used together at Zynga Boston. JIRA is a super-popular bug tracker used by companies like Twitter, eBay, and NASA. The pros of using this software are the thousands of hours of collective experience the team has with it, the powerful reporting and search functionality, and the incredibly large add-on collection available in the Atlassian Marketplace. The con of using JIRA is that it costs dollars. Like, actual money. JIRA was promptly removed from consideration.
Facing the fiscal reality of working for a startup, I found five free and open source trackers that were well reviewed online to compare. First and foremost, I looked for trackers that had the best out-of-the-box features, since I have limited technical abilities and don’t want to rely too much on anyone after the initial setup. After that, I looked for an informative dashboard, in-depth searches, intuitive workflows, issue customization, and the potential to track time and tasks should we ever decide to move away from Trello (which is working just fine for right now). With those prerequisites, the final five ended up being Apache Bloodhound, Bugzilla, Mantis, Redmine, and Roundup.
- Built to be a better Trac
- Serves as a web interface for Git
- Python >= 2.6 and < 3
- PostgreSQL or SQLite DB
- Apache web server
- 3 years user experience
- Many big companies use it (Facebook, NASA… back when NASA was a thing)
- Reports are pretty comprehensive
- Server software usually installed on Linux or Solaris
- Perl <= 5.8.1
- MySQL, PostgreSQL or Oracle DB
- Any server, but Apache 1.3.x or 2.x recommended
- Looks super easy to set up.
- PHP <= 5.2
- MySQL <= 4.1, MS SQL, PostgreSQL, or DB2
- Apache web server
- Recommended by Pikop
- Great Git integration
- Need to create an account
- Ruby 1.8.7, or 1.9.2, or 1.9.3, or 2.0.0, or JRuby 1.6.7, or JRuby 1.7.2
- Rails 3.2.13
- MySQL <= 5.0, PostgreSQL <= 8.2, or Microsoft SQL Server <= 2008 DB
- Need to DL and install to run demo
- Python >= 2.5 and < 3v
- SQLite, MySQL, or PostgreSQL DB
Bloodhound, according to its website, stands “on the shoulders of Trac.” Trac is often listed as a top open source bug tracker and, according to its Wiki, is also used by NASA. Well, how about that? Cool. So, yeah, Bloodhound had the basics of what I was looking for, but didn’t blow me away with anything in particular. I liked that ticket creation was easy and the dashboard was basic yet useful, but nothing really stood out.
Like JIRA, Bugzilla is one of the giants in the bug tracking software game. So much so, that—you guessed it—NASA uses it as well. Ok, this is just getting out of hand. Anyway, I have about three years of experience with a heavily modified version of Bugzilla from my time testing at Harmonix, with only good memories of it. I was always a fan of how intuitive it felt, and having had no QA background before working with it, I found the ramp-up to be very smooth. Pros included the powerful reporting, which in itself included the whine functionality. Whines are reports that get emailed to users at predetermined times. If you want a daily or weekly reminder on how many open bugs there are in your project, it’s a simple task to set it up. One problem we noticed right away, though, was the lack of a default dashboard. After looking around for some extensions, none seemed to hit the spot, so we moved on knowing that if we chose Bugzilla, this would be an issue we’d have to tackle.
I loved Mantis from the moment I first laid eyes on it because I love sherbet and it had all the classics: orange, lemon-lime, watermelon.
Mantis had exactly what we were looking for: a great dashboard, lots of customizable options for issues, strong search and report features, natural fruit flavors. Seriously, Mantis was in the lead for me almost from the moment I dove into the demo.
Redmine also impressed me right away with its simple but thorough design. Right out of the box, it was clear we could easily use this tracker for Features and Engineering tasks, should we decide to consolidate in the future, thanks to the Bug/Feature drop down in New Issues submission, as well as the Gantt chart tab. My Page has your standard dashboard, the Issues tab has all the search options one needs to find exactly what one’s looking for, and optimizing the issue workflow was crazy easy. After some digging, my favorite find ended up being how easy it was to bulk edit via the Context Menu. Find what you need in the Issues tab, check which bugs to edit, and right click to open it up.
Redmine was recommended by a friend of mine who is also in QA, but far more technical than I am. He pitched it as a great way to integrate our version control (Git) with bug tracking, which is something I am incredibly interested in, but have literally no idea how to do. Luckily, our Director of Engineering, Joe Mukai, does, so he spent some time talking it over with the aforementioned friend and gave it a thumbs up. THANKS JOE! At this point, the only thing going against Redmine was that I could find no proof it was ever used by NASA.
Roundup had a tough hill to climb after I spent time with Mantis and Redmine. It became quite clear that while extremely customizable, it would have taken the most work to get it up to speed and match what was offered by Mantis/Redmine right away. To be honest, not a lot of time was spent considering Roundup. Seems like a real good kid, though…
Knowing that no decision was necessarily final, but wanting to avoid rewriting a hundred bugs in the future, I narrowed the field down to Mantis and Redmine. I knew once either was up and running, I’d be able to start creating and assigning bugs and ramp up the team on the process immediately, with no downtime for building extensions or searching for add-ons. On the fence, I followed up with my friend who once again pushed Redmine as a tracker he has twice set up for companies. When I searched my network and came up blank for any personal recommendations for Mantis, my decision was made, with Mantis losing by a neon-colored hair to Redmine.
After three weeks of using Redmine, I feel we made the right choice. If you have any questions about the exciting world of QA, send me an email at firstname.lastname@example.org.