From: Bob Stout 

On Sun, 2 May 2004, Roger Scudder wrote:

> During the time I worked on that project I saw a lot of things that I
> believe contributed to its demise.  The most significant problem IMHO
> was that the programmer who was in the role of designer did not have
> enough experience to tackle the technical problems.  The suggestion was
> made that a consultant be retained to help with the design, but this
> person managed to convince the team that he was up to the challenge.
> Another thing that that had a negative impact was personal ego.  One
> team member seemed to be obsessed with making a name for himself by
> writing super tight super efficient code.  The result was (as I had
> warned it would be) code that suffered so badly from binding that it was
> near to impossible to maintain.  This person ended up having to rewrite
> more than once.  Still another problem was time lost because of people
> failing to properly document changes or failing to document bug fixes. I
> could go on but I think you probably get the idea.
>
> Obviously this project started off on the wrong foot, but I am sure some
> elements of it exist in many other projects to some degree.  My point is
> that the methodology used won't make much difference if certain basic
> requirements are not met.  A team consisting of people who posses
> appropriate experience, who are able to show selfless dedication, and
> who know when it is appropriate to consult with a domain expert, are
> going to be successful regardless of the approach or technique used.

I regard the project I just finished as a really good example of how to do
such a project right.

The project was a 3-D rotational steering tool (directional drilling
control for oil wells). The company had a tool and it mostly worked, but
the firmware was a disaster - if you breathed on it, it broke. Worse yet,
the fellow who wrote it (I can't really say anyone "designed" it)
is an information hoarder who always appears cooperative while providing as
little real information as possible.

The company brought me on contract with the mandate to do whatever it took
to get the thing working and maintainable. Aside from the technical
challenge, the company is typical of many in the oil field services
industry in that firmware engineering takes a back seat to the mechanical
engineers and field personnel. I.e. it's in the electronics and firmware
business because it has to be, but that's not it's business.

Since they wanted it fixed, I started out with an effort improve their
processes and procedures. Using software metrics (which no one had ever
heard of), I showed them why the code was unreliable. Judged by either
lines of code or cyclomatic complexity, the code base was a disaster! The
most egregious example was one function which consisted of 2359 lines (1235
code lines) with a complexity of 190. I told them that most shops using
complexity metrics require justification for single function complexities
above 7.

The next thing I did was introduce them to the wonderful world of software
version control. Their backup and maintenance activities were chaotic, at
best, and they'd been known to have to rewrite firmware that had gotten
lost. I also wrote up a brief document to formalize coding standards.

Next, they expected me to start writing code. Instead, I got everyone
together who knew what the existing tool did and what they wanted it to do
and we hammered out a functional spec. It was a new experience for them,
but they learned from it that not everyone's perceptions and expectations
are the same.

With the functional spec complete, I began the design. What I knew, and
they didn't figure out until much later, was that the new
"firmware" included a fairly complete cooperative multitasking
OS. Using an existing off the shelf RTOS would have been problematic. The
performance requirements were *very* tight - lots of FP math (the 68332
doesn't have an FPU) and the devices we had to use because of the downhole
environment (i.e. hot!) limited us to byte-wide memory access. I wasn't
sure that the overhead of an existing RTOS would work, so I didn't push for
it.

As I mentioned before, the other programmer assigned to work with me on it
initially didn't believe any of the design would work. I knew it would
because I'd used a lots of similar code several years ago writing modem
firmware for Compaq, also using a 68332.

To make a long story short, I finished up this past week. Overall, our
schedule performance exceeded expectations. There was grumbling at first
because they didn't understand my insistence on having a solid design
before I began coding. By now, they realize that the time spent up front on
specification and design paid benefits at the end of the project. Every
aspect of performance of the new firmware is significantly better than the
old firmware (which I expected, but they never even asked for), and
maintenance takes hours instead of weeks. At the tail end of the project, I
captured all my design notes into a detailed design/implementation spec
which can serve as a guide to whoever has to maintain the system in the
future.

Aside from this specific success story, now all the other firmware groups
there are making use of the version control system and the management has a
new respect for the use of formal specification as well as doing real
design rather than hacking code together. When I arrived, all their
firmware development activities were at CMM level 1. Now they're mostly
between levels 3 and 4, with my project a solid 4. What they'll do in the
future, I can't say. But at least they now know there's a better way than
what they did before.

-------------------------------------------------------------
Consulting:
  http://www.MicroFirm.biz/
Web graphics development:
 http://Image-Magicians.com/
Software archives:
  http://snippets.snippets.org/
  c.snippets.org/   cpp.snippets.org/      java.snippets.org/
  d.snippets.org/   python.snippets.org/   perl.snippets.org/
  dos.snippets.org/ embedded.snippets.org/ apps.snippets.org/
Audio and loudspeaker design:
  http://LDSG.snippets.org/   http://www.diyspeakers.net/

--- BBBS/LiI v4.01 Flag-5