1. This forum section is a read-only archive which contains old newsgroup posts. If you wish to post a query, please do so in one of our main forum sections (here). This way you will get a faster, better response from the members on Motherboard Point.

Test Driven Development (TDD) Framework for Embedded Systems

Discussion in 'Embedded' started by moogyd, Apr 23, 2014.

  1. moogyd

    moogyd Guest

    Hi,
    We currently do most (all) our embedeed FW testing on either real HW (if available), using FPGA, or using HW simulator and Verilog models.
    We are looking at performing more testing on the "Host" rather than the target, and since this is new to our group, I am looking for suggestions and comments.
    BTW, we use C, currently we don't use an RTOS, but going forward, this is something that we will be looking at
    Specifically: Do you use a Unit Test Framework, and does it have any features specific to embedded systems?
    Thanks,
    Steven
     
    moogyd, Apr 23, 2014
    #1
    1. Advertisements

  2. moogyd

    Tim Wescott Guest

    I'm surprised you haven't gotten any answers yet.

    I use a unit test framework (cppunit), but it gets used on all of the
    code that doesn't depend on hardware, or hardware behavior. So there's
    lots of code that gets "unit tested" by carefully going through and
    making sure the system works correctly.

    I've never tried it, but I've heard good things about c-unit (cunit?).

    The challenging if you're going to test anything that depends on the
    hardware is making a good behavioral model of that hardware: if your code
    needs to wiggle pin A, see a number appear in register B, then wiggle pin
    C, you're going to have to do a lot of work in the hardware abstraction
    -- and then your unit tests are only as valid as your hardware
    simulation. That, in a nutshell, is why I don't do more testing on the
    hardware.
     
    Tim Wescott, Apr 24, 2014
    #2
    1. Advertisements

  3. moogyd

    Paul Rubin Guest

    I've used different frameworks for different languages. For C, I've
    mostly used DejaGNU which is more of a regression test system than about
    unit testing. Some approaches to unit testing tie in with OOP and
    therefore don't really fit C that well.

    I think test-driven development isn't a good match for C if that's what
    you are considering. TDD emerged from languages like Smalltalk which
    had no static type system but lots of runtime error checks. So most
    coding errors would cause a runtime exception that would crash the
    program, and you'd write enough unit tests to exercise all parts of the
    code, which would usually trigger most of the errors. C has slightly
    more compile time checks but no runtime checks, so errors that get
    through the compiler also have a good chance of not getting noticed by
    simple testing (another aspect of TDD philosophy is to write lots of
    small tests that are very simple).

    If you're doing something serious in C, you have to rely more heavily on
    clean design, careful code reviews, and (lately) fancy static analysis.
     
    Paul Rubin, Apr 24, 2014
    #3
  4. moogyd

    Tom Gardner Guest

    I don't see why you claim that; the problem isn't
    in the way the program is structured, it is in the
    way the execution environment allows silent but
    deadly errors to go unnoticed. And that's just
    as possible with C++ (ugh) or Objective-C (i.e.
    a respectable combination of C and Smalltalk)

    .... but which is strongly typed, unlike C. In Smalltalk and
    Java you simply can't pretend that a Car is a Person (i.e cast
    a Car into a Person).

    TDD became really practical with languages like Java,
    which has both strong run-time typing and a decent
    type system.

    I don't know where you get that idea. For a start, in
    Smalltalk the concept if a "program" is a bit vague - you
    have your code being edited and compiled in an "image", and
    executed by a VM.

    Unless there's a VM error, the nearest to a "crash" is either
    - a "does not understand" message being caught and handled
    by the code in the image being executed by the VM, or
    - your code simply not working as expected

    In other words, the VM+image continues operating normally,
    without error - completely unlike C, of course.

    Smalltalk is neither better nor worse than any other
    language in that respect, since it is dictated by
    budget and skill.

    While both are true, the parentheses aren't related to
    the preceding sentence, of course.

    And most especially a heavily constrained subset of the
    language and library that don't prevent static analysis.
    Same is true for ADA.
     
    Tom Gardner, Apr 24, 2014
    #4
  5. moogyd

    Paul Rubin Guest

    I'm talking about the pattern where you want to test some code that
    talks to the external world, but instead of making i/o calls directly,
    it interacts with an "external world" object that you pass into the code
    (Java applets always work that way). So you can implement both a live
    object that does the real i/o, and a test or mock object that supplies
    canned "external" data and checks the responses. Yes of course you can
    use this approach in C with some contortions, but it's not natural C
    style.
    Nah, it originated in dynamically typed languages and works quite well
    in them (Python, Ruby, etc. have popular TDD frameworks).
    Yes, that would count, normal execution of the function under
    test is stopped.
    The relation is that traditional TDD doesn't attempt to be exhaustive.
    The programmer writes tests to exercises typical cases and also the
    corner cases that come to mind. I don't think JUnit does any automatic
    fuzz testing. Haskell's QuickCheck does that (I don't know if it was
    the first) and it has propagated to some other languages like Erlang.
    True, especially in critical systems that must always work (give right
    answers) for every input, which is probably a more significant issue in
    embedded systems than in (say) typical web applications or compilers.

    More usual programs must never give a wrong answer, but if some weird
    unexpected input ends up making the program fail by printing "error,
    please contact support", that's just an annoyance rather than a
    disaster. Simply using memory-safe languages transforms a lot of
    "potentially disastrous wrong answer" cases into "annoying error
    message" cases, which is a big help with such programs.

    Constraining C is of some help with static analysis but it only goes so
    far. Here is an interesting blog post about why static analysis missed
    the Heartbleed bug, and another about an improvement that the Coverity
    guys are trying as a result:

    http://blog.regehr.org/archives/1125
    http://blog.regehr.org/archives/1128
     
    Paul Rubin, Apr 24, 2014
    #5
  6. moogyd

    Tom Gardner Guest

    I first did that, in C, in (gulp) 1982. It made the embedded
    system very easy to debug by running it in a test harness on
    a PDP11 instead of the embedded Z80.

    It is no more natural or unnatural than any other model
    (either implicit or explicit) in C or any other language.
    All programs contain many different models; the better
    programs make the models explicit.

    TDD was popularised there, but it didn't originate there.
    I've come across many ad-hoc TDD-style frameworks, starting
    with those for pure hardware.

    There are significant qualitative differences. In particular
    you can easily see, inspect and debug the path and objects
    that lead up to the well-contained explicit message.

    Contrariwise, in C you are lucky if you detect that something
    happened at some time in the past, but all useful evidence has
    not been retained.

    Of course. Embedded environments are second-rate by comparison.

    Sure. There are no silver bullets.
     
    Tom Gardner, Apr 25, 2014
    #6
  7. moogyd

    Ed Prochak Guest

    Hi Steven,

    I would suggest the first step for you is changing you software architecture slightly. In any embedded system there is the software that deal with the bit twiddling of the hardware and the application logic that makes the product what it really is. Separating those two aspects is important. So I suggest you design with a hardware abstraction layer and application layer. All the application layer code can be tested on a host. The hardware layer code can also be unit tested if the hardware can be separately simulated (e.g., the communications hardware in the device does not share bits in registers with the motor control).

    You mentioned that you do't yet use an RTOS. Are your programs written in the Grand DO_ALL loop? Long term, I think you will find an RTOS can really simplify your development.

    ed
     
    Ed Prochak, Apr 26, 2014
    #7
  8. moogyd

    Tom Gardner Guest

    I wholeheartedly agree with that.
    For many embedded applications a "full blown" HAL would be overkill.
    Usually all that is needed is a clear distinction between the
    objective/intent (e.g. indicate "error") and mechanism (e.g. set
    bit 5 high to turn on LED). If there is a single lightweight mechanism
    coupling all the mechanisms and objectives, then it will help
    when replacing the test harness on the host with the real logic,
    and replacing the test harness on the target with the application.

    That's a classic build-vs-buy dilemma, and whether it is worth
    it largely depends on what protocol stacks come with the RTOS.

    If you can cast your application into the form of FSMs that
    receive[1] and generate[2] events, then an RTOS is very likely to
    be helpful -- and it will greatly ease your debugging and integration.

    But you knew that, I expect.

    [1] e.g. motor too hot, button pressed etc
    [2] e.g. turn led on, warn operator, switch fan on
     
    Tom Gardner, Apr 26, 2014
    #8
  9. Am 23.04.2014 20:19, schrieb :
    Hello Steven,

    maybe the Unity test framework is what you are looking for. It is a tiny
    test framework for embedded systems and it is completely written in C.

    There was an article on Dr. Dobb's last year:
    http://www.drdobbs.com/testing/unit-testing-in-c-tools-and-conventions/240156344


    Best regards,
    Olaf
     
    Olaf Barheine, May 1, 2014
    #9
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.