How many of your jobs are like this?

Hi:

I was tasked to create a modern upgrade to the central experiment
controller for each of 8 multi-million $$$ research laboratories. They
are still using a DOS based system, except for the first proto that
we've deployed.

I get to make the real-time hardware and software, and the programmer
who worked with the department since the PDP-11 days and developed the
old system gets to make the PC GUI, which has to be backward compatible
with old-style text input files.

The new system has to account for all the lab variations, including one
with a dual experiment necessitating various multiplexers and
complications. And add oodles of new capabilities.

There are no written specs. I poll the scientists to pin down all the
requirements. In the process I suggest to them what we could do, which
they want all of and more. I write down most of the requirements and
set to work. Every once in a while they request new capabilities or
decide to use higher resolution sensors that the programmer promised we
would never need, so my design had better incorporate enough headroom to
be able to adapt without re-spin.

So over the course of several years, while also being the department's
Laser Technologist, completing numerous other electronics projects, and
getting sidetracked by upper management to become an electrical safety
guy and UL inspect 2000 pieces of equipment--which I finally threatened
to leave over and they backed down after I got done with about 20:

I created what I think is an innovative approach to eliminate the old
practice of custom tweaking the real-time code to implement each variant
of the experiment and enable the deployment of a flexible yet standard
solution for all the labs. I chose to implement a dynamically
reconfigurable state machine which outputs sequenced digital waveforms
and pulse generator parameters. All glitch-free, and able to be
reconfigured on the fly while running. Oh, and the state machine
execution engine meets the real-time deadline of the 4.17us period
between 240kHz shaft encoder ticks.


Now that it's done, they don't want to deal with any complexity, don't
want to read any documentation or learn any new concepts (such as a
concept that is central to elementary control theory--the state
machine--and these are all Ph.D. engineers), and the programmer is
having fits because he doesn't like the state machine control model and
is 2nd guessing all of my design decisions because the metaphor doesn't
fit a canonical "programming loop" construct.


Other than that, it's a great job and they provide me with lots of cool
equipment and for the most part let me explore many neat new ideas, so
I'm not complaining. It's just funny (so funny I was up at 3:30am last
night worrying about it all).


Let me guess, this is par for the course?


--
_____________________
Mr.CRC

SuSE 10.3 Linux 2.6.22.17

Mr.CRC wrote:

> Hi:
>
> I was tasked to create a modern upgrade to the central experiment
> controller for each of 8 multi-million $$$ research laboratories. They
> are still using a DOS based system, except for the first proto that
> we've deployed.
>
> I get to make the real-time hardware and software, and the programmer
> who worked with the department since the PDP-11 days and developed the
> old system gets to make the PC GUI, which has to be backward compatible
> with old-style text input files.
>
> The new system has to account for all the lab variations, including one
> with a dual experiment necessitating various multiplexers and
> complications. And add oodles of new capabilities.
>
> There are no written specs.

That is the first big mistake.


> I poll the scientists to pin down all the
> requirements. In the process I suggest to them what we could do, which
> they want all of and more. I write down most of the requirements and
> set to work.

Second mistake is not getting the scientists to agree the written
requirements specification. They should understand that, once written,
agreed and signed off by them, any changes are going to add cost.

> Every once in a while they request new capabilities or
> decide to use higher resolution sensors that the programmer promised we
> would never need, so my design had better incorporate enough headroom to
> be able to adapt without re-spin.

Part of your written specification should be complete interface
descriptions, including resolution ranges and accuracy of translation.

> So over the course of several years, while also being the department's
> Laser Technologist, completing numerous other electronics projects, and
> getting sidetracked by upper management to become an electrical safety
> guy and UL inspect 2000 pieces of equipment--which I finally threatened
> to leave over and they backed down after I got done with about 20:
>
> I created what I think is an innovative approach to eliminate the old
> practice of custom tweaking the real-time code to implement each variant
> of the experiment and enable the deployment of a flexible yet standard
> solution for all the labs. I chose to implement a dynamically
> reconfigurable state machine which outputs sequenced digital waveforms
> and pulse generator parameters. All glitch-free, and able to be
> reconfigured on the fly while running. Oh, and the state machine
> execution engine meets the real-time deadline of the 4.17us period
> between 240kHz shaft encoder ticks.
>
>
> Now that it's done, they don't want to deal with any complexity, don't
> want to read any documentation or learn any new concepts (such as a
> concept that is central to elementary control theory--the state
> machine--and these are all Ph.D. engineers), and the programmer is
> having fits because he doesn't like the state machine control model and
> is 2nd guessing all of my design decisions because the metaphor doesn't
> fit a canonical "programming loop" construct.
>
>
> Other than that, it's a great job and they provide me with lots of cool
> equipment and for the most part let me explore many neat new ideas, so
> I'm not complaining. It's just funny (so funny I was up at 3:30am last
> night worrying about it all).
>
>
> Let me guess, this is par for the course?

Not on my watch. It was in my very early days then I learnt the better way
to organise a project. These days I get to sleep easy at night.

I can recommend reading "Better Embedded System Software" by Philip Koopman.
See http://koopman.us/ for details about the book.

--
************************************************** ******************
Paul E. Bennett...............<email://>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
************************************************** ******************

On Fri, 17 Jun 2011 11:28:16 -0700, "Mr.CRC"
<> wrote:

>Now that it's done, they don't want to deal with any complexity, don't
>want to read any documentation or learn any new concepts (such as a
>concept that is central to elementary control theory--the state
>machine--and these are all Ph.D. engineers), and the programmer is
>having fits because he doesn't like the state machine control model and
>is 2nd guessing all of my design decisions because the metaphor doesn't
>fit a canonical "programming loop" construct.

Let me guess ... you've never dealt with users before. Users almost
never like it when existing processes change and NEVER want to read
documentation.

Moreover, if I understood correctly, you've removed some of the
ability of the users to hack custom applications. That doesn't sit
well with NIH (not invented here) control freaks. It doesn't matter
how clever or easy to use your extension method is, they want the
control that comes from do-it-yourself.

>Let me guess, this is par for the course?

Absolutely.

I used to write industrial QA/QC software that had to have quite
complex control available to supervisory users while presenting only
doorknob level controls to ordinary users and requiring in-application
security to prevent ordinary users from, either inadvertently or
deliberately, screwing up or breaking anything. [Some of the
industries served literally used piece-work day laborers who would
foul up things deliberately to take a break or to bypass QC so as to
get paid more.] At the same time the functions were getting more
sophisticated and supervisory control was getting more complex, the
ordinary user controls had to be secured and idiot proofed, then
imbecile proofed, and finally moron proofed ... and meanwhile the
applications had to get smarter and smarter about realizing and
sounding alarms when the software or hardware was being messed with.

George

Paul E. Bennett wrote:
> Mr.CRC wrote:
>> There are no written specs.
>
> That is the first big mistake.

I'm not sure that in the research environment, there is any other
choice. I mean, I can write some specs for them, such as electrical
specs. But for behavior, there was no way 5 years ago for us to foresee
what experiments would need to be done today. They would simply
complain to the manager that I wasn't serving their needs if I asked
them to specify anything so precisely. They want me to figure out the
precise details of what they want, for them.

>> I poll the scientists to pin down all the
>> requirements. In the process I suggest to them what we could do, which
>> they want all of and more. I write down most of the requirements and
>> set to work.
>
> Second mistake is not getting the scientists to agree the written
> requirements specification. They should understand that, once written,
> agreed and signed off by them, any changes are going to add cost.

They don't care about added cost, since the cost of paying me is already
committed. They only care that when they want it to do something
different from what they said they wanted it to do last time, that it
happens as soon as possible.

>> Every once in a while they request new capabilities or
>> decide to use higher resolution sensors that the programmer promised we
>> would never need, so my design had better incorporate enough headroom to
>> be able to adapt without re-spin.
>
> Part of your written specification should be complete interface
> descriptions, including resolution ranges and accuracy of translation.

Well the way it actually works is that I ask them what their current
system does, what they think needs improvement, etc. Then I have to
become familiar with their work. I have to get in their experiment and
understand it, get familiar with the direction of their work, then begin
to make best guesses at what they probably will want to do in the future.

My experience in this research environment so far is that every time I
am told to just implement the immediate need, that it is a big mistake
and it would have been better to spend a little more time to think
through a more forward reaching solution.

Every time I think ahead, they wind up actually doing what I thought
they would do.

>> Let me guess, this is par for the course?
>
> Not on my watch. It was in my very early days then I learnt the better way
> to organise a project. These days I get to sleep easy at night.

There is no organizing here. Most of the time, I am it, the only
electronics person in a department of mechanical engineers. This time I
have to work closely with the programmer, who originally wanted the
simplest thing possible, which would have turned out useless the moment
it was committed to HW.

> I can recommend reading "Better Embedded System Software" by Philip Koopman.
> See http://koopman.us/ for details about the book.

Thanks for the reference and the input.



--
_____________________
Mr.CRC

SuSE 10.3 Linux 2.6.22.17

George Neuner wrote:
> On Fri, 17 Jun 2011 11:28:16 -0700, "Mr.CRC"
> <> wrote:
>
>> Now that it's done, they don't want to deal with any complexity, don't
>> want to read any documentation or learn any new concepts (such as a
>> concept that is central to elementary control theory--the state
>> machine--and these are all Ph.D. engineers), and the programmer is
>> having fits because he doesn't like the state machine control model and
>> is 2nd guessing all of my design decisions because the metaphor doesn't
>> fit a canonical "programming loop" construct.
>
> Let me guess ... you've never dealt with users before. Users almost
> never like it when existing processes change and NEVER want to read
> documentation.

Heh heh. And then they will also complain if there isn't detailed
enough documentation when they get messed up to the point where they
finally decide to read it.

I'm more used to giving my users hardware. Software or a system with
complex user-configurable behavior is a new thing.

> Moreover, if I understood correctly, you've removed some of the
> ability of the users to hack custom applications. That doesn't sit
> well with NIH (not invented here) control freaks. It doesn't matter
> how clever or easy to use your extension method is, they want the
> control that comes from do-it-yourself.

Perhaps you misunderstand a little. The department is quite rightfully
concerned that when their programmer, who has specialized knowledge of
engine research and the ability to integrate that into his programming,
knowledge that is going to be very hard to replace, that they will be in
a quandry.

You see it is the programmer, not the scientists, who customizes each
lab control system to the details of the lab. This is done at the code
level, not just configuration parameters.

I have been reading a lot of the technical notes and papers here:

http://www.stateworks.com

and some other literature about reconfigurable state machines, and have
become convinced that it is better to implement flexible system behavior
by changing configuration information, which instructs an execution
engine to exhibit the behavior, rather than hard coding the behavior.

The scientists are not going to be able to customize embedded real time
code to do what they want once the programmer retires.

The system that I have designed is capable of exhibiting all of the
behavior of all the separately customized control systems, and then much
much more that the current systems cannot do, without any hard coding of
the behavior. So I have achieved the goal of vastly more capability
with standardized code for all labs.


>> Let me guess, this is par for the course?
>
> Absolutely.
>
> I used to write industrial QA/QC software that had to have quite
> complex control available to supervisory users while presenting only
> doorknob level controls to ordinary users and requiring in-application
> security to prevent ordinary users from, either inadvertently or
> deliberately, screwing up or breaking anything. [Some of the
> industries served literally used piece-work day laborers who would
> foul up things deliberately to take a break or to bypass QC so as to
> get paid more.] At the same time the functions were getting more
> sophisticated and supervisory control was getting more complex, the
> ordinary user controls had to be secured and idiot proofed, then
> imbecile proofed, and finally moron proofed ... and meanwhile the
> applications had to get smarter and smarter about realizing and
> sounding alarms when the software or hardware was being messed with.
>
> George

Interesting experience. Thanks for your response.


--
_____________________
Mr.CRC

SuSE 10.3 Linux 2.6.22.17

Hi Paul,

On 6/17/2011 2:49 PM, Paul E. Bennett wrote:
> Mr.CRC wrote:

>> Let me guess, this is par for the course?
>
> Not on my watch. It was in my very early days then I learnt the better way
> to organise a project. These days I get to sleep easy at night.

<grin> My favorite is when you're trying to nail down a
spec and grilling the client on what he wants to happen
in <various_circumstances>.

Typically, they'll respond with "I don't care..." (i.e.,
*you* figure it out).

While taking notes on these exchanges, I have learned how to mumble
these words (or equivalent) sotto voce:

"When <various_circumstances>, short out the power supply, spin
down the drives without retracting the heads, erase all of
memory..."

(doing so with a straight face -- "professionally" -- takes *lots*
of practice! :> )

The sign of sheer terror on the client's face makes it well worth
the effort! *And*, they are usually *very* careful not to utter
the 'I don't care' phrase, again!

(there *are* some rewards for being in this business!)

Don Y wrote:
> <grin> My favorite is when you're trying to nail down a
> spec and grilling the client on what he wants to happen
> in <various_circumstances>.
>
> Typically, they'll respond with "I don't care..." (i.e.,
> *you* figure it out).
>
> While taking notes on these exchanges, I have learned how to mumble
> these words (or equivalent) sotto voce:
>
> "When <various_circumstances>, short out the power supply, spin
> down the drives without retracting the heads, erase all of
> memory..."
>
> (doing so with a straight face -- "professionally" -- takes *lots*
> of practice! :> )
>
> The sign of sheer terror on the client's face makes it well worth
> the effort! *And*, they are usually *very* careful not to utter
> the 'I don't care' phrase, again!
>
> (there *are* some rewards for being in this business!)


That's very funny! I'll keep that in mind.



--
_____________________
Mr.CRC

SuSE 10.3 Linux 2.6.22.17

Don Y wrote:
> Hi Paul,
>
> On 6/17/2011 2:49 PM, Paul E. Bennett wrote:
>> Mr.CRC wrote:
>
>>> Let me guess, this is par for the course?
>>
>> Not on my watch. It was in my very early days then I learnt the better
>> way
>> to organise a project. These days I get to sleep easy at night.
>
> <grin> My favorite is when you're trying to nail down a
> spec and grilling the client on what he wants to happen
> in <various_circumstances>.
>
> Typically, they'll respond with "I don't care..." (i.e.,
> *you* figure it out).
>

Then I write them out and send it to them.


> While taking notes on these exchanges, I have learned how to mumble
> these words (or equivalent) sotto voce:
>
> "When <various_circumstances>, short out the power supply, spin
> down the drives without retracting the heads, erase all of
> memory..."
>
> (doing so with a straight face -- "professionally" -- takes *lots*
> of practice! :> )
>
> The sign of sheer terror on the client's face makes it well worth
> the effort! *And*, they are usually *very* careful not to utter
> the 'I don't care' phrase, again!
>
> (there *are* some rewards for being in this business!)


Me, in a board room, nice dark suit and tie (wife picked that out).
Giving a presentation with lots of big ticket cost factors in there,
machine amortizations and all that fun stuff. At the end all the other
suits seemed content with what was presented, the mood became a bit more
loose, everyone had their thoughts on lunch, the smells of which were
already wafting in. Then one guy asked "What's your confidence level in
those numbers?" ... "Ahm, plus minus 3db" ... silence, you could almost
here a pin drop ... then thundering laughter.

During lunch the CEO asked me never to pull one like that again, almost
gave him a cardiac event he said.

--
Regards, Joerg

http://www.analogconsultants.com/

"gmail" domain blocked because of excessive spam.
Use another domain or send PM.

On 6/17/2011 4:41 PM, Mr.CRC wrote:

>> (doing so with a straight face -- "professionally" -- takes *lots*
>> of practice! :> )
>>
>> The sign of sheer terror on the client's face makes it well worth
>> the effort! *And*, they are usually *very* careful not to utter
>> the 'I don't care' phrase, again!
>>
>> (there *are* some rewards for being in this business!)
>
> That's very funny! I'll keep that in mind.

**Don't** do it unless you have a really good feel for your
customer/client!

With *good* (i.e., "rational") clients, this can lead into
a sober discussion along the lines of:

"Well, Bob, how would you like me to go about figuring out
what *should* be done in these situations? I can take
the easy way out (for me) -- like 'reset the device and
start over from scratch'. That will keep your development
costs down -- anything you can't foresee *here*, *now*
won't affect the final product's design or its cost.
OTOH, your users might not like the way the device behaves
in those cases. Especially if this means they lose 'something'
(time, data, test results, product, etc.).

If, OTOOH, you just want me to 'do whatever it takes' to get
the *right* solution, then your costs (and timetable) become
unbounded -- but, we can be reasonably assured that the
end user will be happy with the result.

Sure, I can *call* you each time I encounter one of these,
'what-do-you-want-to-do-if' scenarios. But, will you be
able to give me a quick, well thought-out reply? Who'll
bear the cost of documenting that? Do I have to update my
time/cost estimates each time this happens? Who will bear
the cost of *that*?? Or, do I have to put whatever I was
working on *aside* and try to *find* something to keep me
busy (progressing towards a final product) while you think
about the 'what-if'?

Lastly, do you think either of us are really going to be
*happy* working like that?

So, why don't you guys give this a bit more thought and
flesh out those areas that you know *I* haven't yet
discovered (since you guys *should* know more about this
than me!) and we can look at the project again, later,
if it still makes sense (you may discover that there is some
huge GOTCHA that just makes this totally impractical).

*Or*, you can hire me on an open-ended, time & materials
basis to research your project, the needs of the targeted
users, etc. and *write* a specification for you. You can
then take this and either ask *me* to bid on it. Or, pass
it around as an RFQ and get someone *else* to bid on it..."

I.e., this puts the comment into a rational perspective.
"I'm not trying to be 'cute'... or 'a smart-ass'. This is
what the consequences of unknowns in your specification
will be. How would you like to resolve them so we can
CONTRACTUALLY agree how to proceed?"

Hi Joerg,

On 6/17/2011 5:33 PM, Joerg wrote:

>> <grin> My favorite is when you're trying to nail down a
>> spec and grilling the client on what he wants to happen
>> in<various_circumstances>.
>>
>> Typically, they'll respond with "I don't care..." (i.e.,
>> *you* figure it out).
>
> Then I write them out and send it to them.

I'll do that -- *if* they give me a contract to *write*
that document!

>> While taking notes on these exchanges, I have learned how to mumble
>> these words (or equivalent) sotto voce:
>>
>> "When<various_circumstances>, short out the power supply, spin
>> down the drives without retracting the heads, erase all of
>> memory..."
>>
>> (doing so with a straight face -- "professionally" -- takes *lots*
>> of practice! :> )
>>
>> The sign of sheer terror on the client's face makes it well worth
>> the effort! *And*, they are usually *very* careful not to utter
>> the 'I don't care' phrase, again!
>>
>> (there *are* some rewards for being in this business!)
>
> Me, in a board room, nice dark suit and tie (wife picked that out).
> Giving a presentation with lots of big ticket cost factors in there,
> machine amortizations and all that fun stuff. At the end all the other
> suits seemed content with what was presented, the mood became a bit more
> loose, everyone had their thoughts on lunch, the smells of which were
> already wafting in. Then one guy asked "What's your confidence level in
> those numbers?" ... "Ahm, plus minus 3db" ... silence, you could almost
> here a pin drop ... then thundering laughter.

"Well, the usual engineering estimates typically require you
to double the quantity and step up to the next higher unit.
So, 2 days would become 4 weeks; 3 weeks, 6 months; you get
the drift...". Let them take the estimate you just gave them
and go through this little exercise :>

Again, the key to pulling this off is to maintain a straight
(poker) face.

> During lunch the CEO asked me never to pull one like that again, almost
> gave him a cardiac event he said.

[<grin> I was bringing up a processor some years ago. Device
consumed 500W off the -5.2V. I.e., you could put a screwdriver
across the power rails and the power supply wouldn't *blink*!
I was plugging in a component with power still on (because cycling
power was a long, drawn out process). I was in a pretty good mood
because I was almost done and was 'on a roll'. So, I quipped,
"Watch me *fry* this sucker!" And, as luck would have it, accidentally
shorted the ground plane to the power pin *through* the lead of
the device I was installing. Of course, the lead simply "ceased
to be" -- accompanied by a nice little "crackle" and flash of
light! *I* was startled. My boss was *livid* -- "You did that
INTENTIONALLY!!" <shrug> Some folks just have no sense of humor...]

Getting back to your meeting...

Sometimes "comic relief" can help bring about meaningful discussions.
*Nobody* wants to voice the fears that EVERYONE shares in these
meetings. And, no one wants to call anyone a liar. But, *everyone*
knows about 'the best laid plans...'

If you have people genuinely committed to making a project
work present, this is a great opportunity for them to be
candid about their reservations... things you might not have
seen in the RFQ, things *they* might not realize as consequences
of some of their requests, etc.

At the very least, it helps build a rapport between the parties
so there is less likelihood of folks distancing themselves
from a failing project, fingerpointing, etc.

[BTW, decided *not* to get the heavy cream. I figure 3500
'surplus' calories was a wee bit more than I could afford.
I'll make gelato, instead -- and only have to exercise *half*
as much! :-/ ]