Rally Integration Toolkit for Ruby

This page is meant to be a resource for myself as I work with the API to answer questions about our many ongoing projects.

Why?  When one could use Rally for such things?

In early 2007, Rally Software introduced a Ruby-based toolkit for accessing information stored in their Agile project management database.  Follow the toolkit link for examples of what’s possible.  Among them you’ll find scripts that generate Excel spreadsheets, PDF reports and more.

Rally does many things well, though among them you will not (yet!) find project visualization in stakeholder-digestible fashion.  One my goals for effective communication with stakeholders is to show trends such that surprises are minimized.  In a 6 month release cycle, engineers and project managers may smell a missed date 2 months in though stakeholders only find out in month 4 when the project moves beyond the tipping point (i.e. some arbitrary classification of release readiness “tips” from one grade/color/ranking to the next).  It’s my contention that this is due to a lack of empirical data on behalf of the project manager / SCRUM master.

Gathering the data I need via the Rally API is what I’m working on and I’ll be logging any gotchas to this page.  I apologize in advance for the lack of “flow” through this document – it’s more of a reference than a tutorial.

Getting Started:  Using A ‘Logger’ To Debug Queries

Before you do anything else, do this.

As discussed in the Rally API docs for RestQuery, usage of a logger is very much recommended.  More than once I’ve looked at the raw dump of query results when the output was not what I expected.  For example, using the XML output of a query, I was able to identify that I was getting duplicate object IDs when I was expecting uniques.

Here’s how:

require ‘logger’

my_logger = Logger.new STDOUT

slm = RallyRestAPI.new(
:base_url => “https://rally1.rallydev.com:443/slm”,
:username => “…”, :password => “…”,
:logger => rally_logger)

See the full docs for Logger here at ruby-doc.  On a side note, Logger was co-written by Hiro Nakamura (no, not that one, though at least a bit funny…).

I recommend creating a separate logger for use by Rally’s API only, and point that to something like rally_rest.log.  “tail -f” this file in a separate tab while working on your code so you can always see the debug output scroll by.

Query Block: Use of Scoped Variables Produces Strange Results

This query asks Rally for all stories in the specified iteration for the specified project.  Think of it as SELECT * FROM stories WHERE project = ‘MyProject’ AND iteration = ‘2’.

query_result = slm.find(:hierarchical_requirement, :fetch => true, :project => @project) {equal :iteration, @iteration}

This query produces strange results.  The correct number of stories are returned, though the story objects themselves are completely bunk.  They consist of only a name; no release or iteration data is available through the typical method_missing-based query (story.release, story.name, etc.).

How to fix this?  Remove the “@” in front of iteration.  Using any other type of variable (e.g. contents of a hash, etc.) either produces errors or other strange results.  I’m still very new to Ruby, so I’ll need to do a bit of research to understand why this is.  Interestingly enough, the @project parameter is understood correctly by the query engine.  I’m going to guess that this is due to variable visibility of the block, though I’ll leave this mystery to those more versed in Ruby than myself!

This took me 2 hours (!) to figure out.  Better to learn from my mistakes ;-)

Query Block: No Results When Expected

This will likely be where you spend most of your time when working with the API – debugging query results (or lack thereof!).

Notice anything strange about the query block iteration field I’m using?

query_result = slm.find(:release_cumulative_flow_data, :order => :creation_date) do
equal :release_object_i_d, release.object_i_d
equal :creation_date, iteration.iter_start_date
end

The highlighted field is not a valid member of the Iteration object.  The real name is start_date.  Unfortunately, this is swallowed by the API.  I suspect better handing in method_missing would be possible to throw an exception here?  I’m not sure.

How Do I Work With Nested Projects and Story Queries?

Let’s say you have a parent project with a few child projects.  When you create a new release, create it in the parent project and identically in the children (or just click the “Batch create” checkbox when creating a new iteration).  Releases aren’t shared from parent projects, so this step is necessary or else releases are invisible when scoped to the child (i.e. your current project is set to a child).

Yes, Rally’s support for nested projects still has a ways to go ;-)

When querying for stories (:hierarchical_requirement) you only need a single query to retrieve all of the stories.  A ‘handle’ to any of the releases will produce the same output.  So to get all stories across all children:

query_result = slm.find(:hierarchical_requirement, :fetch => true, :project => parent_project) {equal :release, release}

This works because of the implicit parameter :project_scope_down => true.

How Do I Work With Nested Projects and Iteration Queries?

When attempting to isolate nested project iterations, you may have to be a bit more specific in your queries.

If your parent project iterates along with your child projects, and you want to locate iterations only in the parent project, be sure to use a query like this:

query_result = slm.find(:iteration, :project_scope_down => :false, :project => parent_project) {equal :name, “2008-November”}

By augmenting your query with the additional scope down parameter, you’re telling Rally not to include any child iterations.

Conversely, when attempting to isolate child project iterations:

query_result = slm.find(:iteration, :project_scope_up => :false, :project => parent_project) {equal :name, “2008-November”}

The details of this can be found here, in the Rally API docs for RestQuery.

Why Aren’t Epic Stories Returned?

By design!  My story queries always specify at least :project and one of :release or :iteration.  From this screenshot in Rally, you can see that both of these attributes are pulled up from child stories.

You cannot directly modify either the :release or :iteration attributes of a story with children, so returning them in a query like this wouldn’t be accurate – the EPIC itself it not committed to in the iteration for the release, so it shouldn’t be returned.

“Flow Data” Snapshot Time and Frequency

Courtesy of Bob Cotton here:

IterationCumulativeFlowData (and ReleaseCumulativeFlowData) and created every day at midnight (relative to your workspace timezone) for every iteration (and release) in your workspace. There are none for “today” or now.

These snapshots are only taken in the range defined by the iteration.  For example, one of our iterations ended on December 12.  Asking for an ICFD with a creation_date of December 13 will return nothing!

“Of course!  Why is this a problem?”

If you wait until your next sprint planning meeting to “clean up” your previous iteration (a random task here or there is split into a new story, etc.) then the snapshot may not contain everything you’d like.  From a SCRUM perspective, everything should be complete (or moved out) of the iteration by the time it’s ended, so a purist would shake their head at the way we’ll sometimes split and carry a story forward.  I can’t use ICFDs because of this.  I construct a detailed query for hierarchical requirements (i.e. user stories) for each iteration: assigned to that iteration, marked complete or accepted, etc.

Keep in mind the phrase “relative to your workspace timezone” because it means the CreationDate attribute of the FlowData objects will be offset.  For example, I’m Pacific (-8) so my FlowData objects will have a timestamp of something like: 2008-11-27T08:00:00.000Z.  What’s unfortunate is that the start dates of your releases and iterations do not take timezone into account.  This means you need to do a little Ruby trickery to translate the date you get from your releases and iterations into a date consumable by the flow data queries.

Rally’s REST API and Time-based Queries

When querying for ReleaseCumulativeFlowData object created on a certain day, you must specify a timestamp.  For my purposes, this is the start date of each iteration (i.e. a release burn down chart).  When querying for an iteration start date, Rally gives you a date at 12:00 midnight, for example:

2008-11-27T00:00:00.000Z

The timestamp for flow data objects takes your timezone into account, so the flow data object on the same day will have a CreationDate of:

2008-11-27T08:00:00.000Z

How does one manipulate time?

# Calculating release size
iter_start_time = ParseDate.parsedate(r_iteration.start_date)
iter_start_time = DateTime.new(iter_start_time[0], iter_start_time[1], iter_start_time[2])
iter_start_time = iter_start_time + 8.hours

Now you have your DateTime object, though you still need to format it in a Rally-consumable fashion.

Update your /config/environments/date_formats.rb with:

ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.merge!(:rally_timestamp => "%Y-%m-%dT%H:%M:%S.000Z")
ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(:rally_timestamp => "%Y-%m-%dT%H:%M:%S.000Z")

Restart your server and when specifying the time:

equal :creation_date, rally_date.to_s(:rally_timestamp)

There you go… whew ;)

Error Message: “Not authorized to read”

If you receive this error:

<Errors>
  <OperationResultError>Not authorized to read: Project 167543189</OperationResultError>
</Errors>

It’s because you’re using a non-admin account to access a closed project (among other things).  When iterating over a parent project’s children, you’ll receive this message just by using the .each iterator.  To get child projects you need to create a query to retrieve them.

Rally Support: IterationCumulativeFlowData

From Rally Support (thanks to Michael, who was very patient and has taken the time to respond in much detail to my many queries):

The IterationCumulativeFlowData will provide the Total of the points for each status on a given day.

So in your scenario below the 15 point would be added to any other Accepted status points that were already collected.  Therefore if 0 points were in Accepted State on 12/11 then the card_estimate_total for the 12/12 card of the Accepted State would be 15, but if there were 12 points already in Accepted State on 12/11 and 15 points were Accepted on 12/12, then the card_estimate_total for the 12/12 card of the Accepted state would be 27.

So it is the Total points in the given state when the collection is run.  Hopefully that helps explain what is going on.  So it could give you what you need as long as people are good about making sure the States are correct each day.

Rally Support: ReleaseCumulativeFlowData

My question:

Is it possible to query Rally for historical data?  For example, I want to know how many points were in the release before each of the sprints began.

Rally Support (thank you David!) :

I think you should be able to find what you are looking for in the Release Cumulative Flow object. Data collection runs at the end of each day and collects totals by ‘card state’ which is the story state. If you get the object ID of your release, you can then query on this object with that OID and sort by date and then sum up the points for the various states for the first day. (Guessing most will be defined or in backlog, but maybe some industrious developers were already in progress or completed..)

In Closing…?

Historical data in Rally is only as valuable as the effort you put towards maintaining it.  You don’t have to get your estimates perfect and you don’t have to get your story size perfect though what should consider doing is something.  When the pressure is on and you require historical data (velocity especially) to make snap decisions, you’ll wish it was there!

While working with Rally (and this API) can sometimes be frustrating, you will not find more supportive individuals and a more supporting company.  Support replies within 24 hours, they log (and track) features and bugs on your behalf and they host a terrific, active community over at Agile Commons – a can’t miss for fans of Agile development.

Drop me a note if there are any questions and I’ll update this reference if I can!