I recently worked on a project which required extracting bug details from a Bugzilla database; we were attempting to reproduce the results of an academic paper in which the authors attempted to establish a link between object-oriented software metrics and program buginess. Bug data was obtained from a Bugzilla database to determine the buginess of a class. The original authors obtained a copy of the entire bug database and ran queries against it directly. We would not have been able to get a copy of the entire database, but fortunately it is now available on-line. We started by using the Bugzilla web search interface to get a list of bugs for the time period covered in the original research. We then fed that list of bug IDs into the script below to get the details we needed: which files, or classes, were associated with each bug.

This demonstrates the power of Unix shell scripting. Powerful shell scripting is one of the many reasons why I stay away from the Windows environment; awk, grep, sed, tr, and similar commands make it possible to perform some relatively powerful data manipulations quickly.

#!/bin/sh
#
# bugdetails_from_ids.sh
# Christopher Stoll, 2013-11-26
#
# Get Bugzilla bug patches given the bug IDs
#

if [ ! -e "$1" ]; then
	echo "Usage:\n $0 bug_ids-one_per_line.txt"
	exit
fi

echoerr() { echo "$@" 1>&2; }

# read bug IDs into an array
declare -a bugids=(`cat $1`)

((i=0))
while [ ${bugids[i]} ]; do
	bugid="${bugids[i]}"
	if [ ! -z "$bugid" ]; then
		#echoerr "~~~ $bugid"

		bugdetail=""
		# get the bug details page
		bugdetail=`curl -fLs --max-time 60 \
			https://api-dev.bugzilla.mozilla.org/latest/bug/$bugid/attachment`

		if [ ! -z "$bugdetail" ]; then
			# They give us some nice JSON,
			# which we are going to just hack apart
			#  convert commas to newlines
			#  find lines which contain "ref" (links to the attachments)
			#  grab the http and the rest of the url
			#  remove quoation marks
			atchids=()
			atchids=(`echo "$bugdetail" | \
				tr "," "\n" | \
				grep \"ref\" | \
				awk -F: '{print $2":"$3}' | \
				tr -d '"'`)

			((j=0))
			while [[ ${atchids[j]} ]]; do
				atchid="${atchids[j]}"

				if [ ! -z "$atchid" ]; then
					#echoerr "~~~ $bugid ~~~ $atchid"

					atchdetail=""
					# get the attachment details page
					atchdetail=`curl -fLs --max-time 60 \
						"$atchid?attachmentdata=1"`

					if [ ! -z "$atchdetail" ]; then
						# They give us some nice JSON, again we hack
						#  convert commas to newlines
						#  find lines which contain data
						#   this is the patch data
						#  grab the actual data
						#  remove quoation marks
						#  decode the data from base64
						patchdata=`echo "$atchdetail" | \
							tr "," "\n" | \
							grep \"data\" | \
							awk -F: '{print $2}' | \
							tr -d '"' | \
							base64 --decode`

						atchfile=`echo "$patchdata" | grep +++`
						echo "$atchfile" | sed -e 's/^/'$bugid'/'
					fi
				fi

				sleep 1
				((j=j+1))
			done
		fi
	fi

	sleep 1
	((i=i+1))
done

Tags: Bugzilla Code Shell Script

Published: 2014-01-01