1. This forum section is a read-only archive which contains old newsgroup posts. If you wish to post a query, please do so in one of our main forum sections (here). This way you will get a faster, better response from the members on Motherboard Point.

LPC quad UART garbling 8th bit in TX data - difficult to reproduce -any ideas?

Discussion in 'PC Hardware' started by Frantisek.Rysanek, Dec 21, 2009.

  1. Dear Everyone,

    apologies for cross-posting, I'm wondering if some of the veterans of
    10-15 years ago are still reading these USENET groups...
    I'm sending this problem description just in case the weird symptoms
    ring a bell with someone - if someone happens to have a past
    experience that might resemble my symptoms.
    I will deliberately not name the vendors involved - we are not sure
    exactly where the problem is.

    Strange as it may seem to the majority PC user nowadays, in our
    industrial process control practice we're still using RS232 and its
    flavours RS485, RS422. The problem at hand occurs in a [brand
    censored] panel PC (that's an LCD with an embedded motherboard in a
    flat wallmount box), that's based on an Intel Atom, coupled with the
    i945GSE chipset (including some ICH7 flavour for a south bridge). The
    PC doesn't seem to contain a full-fledged SuperIO chip (no floppy, no
    PS2, no LPT/joystick, just USB) and its three serial ports are
    implemented using a [brand censored] quad UART chip, that connects to
    the ICH via an LPC bus. The quad UART chip has some configurable
    "industrial" features, such as TEMT bit exported to an outer modem
    control signal (for HW-based RS485 RX/TX steering) and something
    called "9bit mode".

    Under some circumstances, for an unknown reason, the quad UART tends
    to produce garbled TX data. You open the HyperTerminal, select a COM
    port, configure it for 9600 8N1, without flow control. If you send a
    string of e.g. the ASCII character "a", the 8th bit in every *second*
    character is inverted. Bit 8 in the RS232 character = the last before
    the stop bit = the MSB. One character okay, another character garbled.
    In HyperTerminal, it looks like a string of good characters mixed with
    non-ascii garbage. An interesting feature seems to be, that it does
    this with *any* character you can type on the keyboard - any letters
    or numbers. If you type different characters in a sequence, the
    garbage doesn't occur. If you keep sending a single character, it
    starts producing garbage. It doesn't matter if you type the characters
    isolated (so that the FIFO is empty all the time) or if you send a
    string via the clipboard, so that the characters get "clocked out"
    back to back (and buffered in the UART's 16byte FIFO in the process).

    The waveform is fairly clear, either the bit is there nice and clear,
    or it's completely absent - there are no glitches or weird
    malformations. There's no doubt that the garbage is coming out of the
    UART's "trasmitter shift register". Observed with an oscilloscope
    straight at the UART's TTL level output, before RS232/485 drivers,
    with nothing attached to the RS232/485 ports (no load on the drivers).
    Observed on port 1 and 3 of the quad-UART chip. Unfortunately I don't
    have an LPC bus analyzer to check if perhaps the data is coming
    garbled from the ICH. Makes me wonder how come that everything else
    works just fine, all the addresses and data on the LPC bus - just that
    bit #8 gets garbled. Otherwise it might be attributed to interference
    from WiFi or BlueTooth (both in the box).

    Another problem is that the symptom tends to go away if you start
    looking too hard or poking around. On some machines it seems to appear
    after a cold boot, after a period of inactivity. On other machines, it
    starts after some flawless production runtime. It definitely seems to
    go away if you switch the baud rate away from 9600 and back to 9600.
    And it doesn't come back after the next reboot...

    An interesting aspect is, that it only occurs on Windows XP Embedded.
    If the problem occurs, and I reboot to Linux (via Etherboot), I cannot
    reproduce it with Minicom or other serial TTY software that I have. If
    I reboot back to XPe on a CF card, you bet it's there. This is a known-
    clean system with only the basic drivers: a stock Windows serial.sys,
    and the next closest driver to RS232 is a Penmount touchscreen driver
    (never a problem with that one, sitting fixed on its own port). I've
    tried with a pristine system, before the visualization app gets
    installed (the production app that works with the COM ports), and I
    cannot imagine a filter driver sitting on the COM ports - in the first
    place I haven't installed any, in the second place I don't know what
    purpose it might have.

    I'd really love to know where the gremlins are hiding this time :)

    If you've read this far, thanks a lot for your attention.

    Frank Rysanek
     
    Frantisek.Rysanek, Dec 21, 2009
    #1
    1. Advertisements

  2. Frantisek.Rysanek

    Franc Zabkar Guest

    What happens if you send a string of genuine 8-bit characters, eg
    extended ASCII? This will force Windows to "think about" the 8th bit,
    whereas anything you type at the keyboard will usually be 7 bits.

    - Franc Zabkar
     
    Franc Zabkar, Dec 21, 2009
    #2
    1. Advertisements

  3. Frantisek.Rysanek

    whit3rd Guest

    .... using a [brand censored] quad UART chip, that connects to
    Well, there's two hints: first, it doesn't happen in Linux. Second,
    it
    'comes and goes'.

    Alas, that leaves two possibilities: a bit of background software that
    reinitializes the UART registers and is unaware of the FOUR ports
    that are there, which inadvertently changes your '8n1' settings
    but which changes it back to the correct value later. It would
    be rational to suspect other serial processes (IRDA as well
    as COM ports). You've already looked there, though.

    And, perhaps there's a bad bypass capacitor near the UART, and
    it allows the power to warble enough to change a logic
    threshold. That would be heat-related, you might be able to
    use a hot-air gun and create the symptom.
     
    whit3rd, Dec 21, 2009
    #3
  4. Frantisek.Rysanek

    tlvp Guest

    You speak of opening "the HyperTerminal" -- is that the terminal app
    that's part of Win XP? If so, be aware that it itself is flawed, at
    least in my experience: when I was using it to capture data from an
    RS-232 serial line communications link, it would habitually and
    consistently buffer its capture contents so badly that scrolling
    back to earlier parts of the capture, that *had* looked OK at first,
    showed very badly garbled contents.

    My solution was to replace it with some other Terminal app, coming
    from the public domain, that turned out *not* to manifest such issues.

    No guarantee that that's where *your* issues arise, of course, but as
    the issue arises under XP and not under Linux, it's a thought ... .

    HTH; and cheers, -- tlvp
     
    tlvp, Dec 22, 2009
    #4
  5. Frantisek.Rysanek

    stratus46 Guest

    Is is possible the send or receive is set to 7 bits + parity and the
    other set to 8 bits? that would mess up the 8th bit on 50% of the
    possible characters.

     
    stratus46, Dec 22, 2009
    #5
  6. You have a "stock Windows serial system"? No Windows updates, no
    driver updates?

    I'd suspect a problem right there. Go ahead and try *ALL* the updates.
    And since serial support is not Windows strong point, and especially
    since Hyperterminal is an incredibly limited, try installing and using
    the 'Putty' freeware, which has decent serial support and vastly
    better terminal emulation.

    Given the concatenation of somewhat oddball chipsets in this system,
    and the claim that it works properly under Linux, I'd suspect driver
    issues, particularly flow control handling. Are your serial devices
    actually wired correctly for hardware flow control? Many well-meaning
    but clueless engineers get the RS-232 signals for different devices
    miswired, and never notice because they've mishandled flow control and
    don't use it much.

    You should also definitely state which Windows and Linux operating
    systems you're using.
     
    Nico Kadel-Garcia, Dec 22, 2009
    #6
  7. Frantisek.Rysanek

    Andy Guest

    In hyperterminal there is an option to treat incoming messages as 7
    bit ascii. It sounds like you are getting 7 bits. Have you tried
    this?
     
    Andy, Dec 22, 2009
    #7
  8. Frantisek.Rysanek

    Ian Malcolm Guest

    Hyperterminal is known to garble *RECEIVED* characters under certain
    circumstances if the first three characters recived are identical. This
    fault, once triggered, persists until the application is exited. I wonder
    if it is possible for the same bug to also affect transmit?

    Please try a 3rd party terminal and see if the problem goes away...

    Also censoring the uart chip brand and number will not lead you to any
    answers...
     
    Ian Malcolm, Dec 22, 2009
    #8
  9. Frantisek.Rysanek

    Bob Larter Guest

    It sounds to me like the problem is an interaction between the standard
    serial driver & the extra features of the UART. Try contacting the
    manufacturer of the UART, & see if they know about this 'bug' & have a
    workaround, or an updated serial driver. What UART is it?
     
    Bob Larter, Dec 23, 2009
    #9
  10. Dear gentlemen,

    many thanks to all of you who responded :)

    I'm delighted to see that the USENET is still alive, and that it's
    sometimes the same
    people responding to me, who used to respond ~10 years ago...

    And, maybe special thanks to Ian Malcolm - his bug description is the
    closest to mine :)
    If I exit Hyperterminal and start it again, the problem is typically
    still there.
    It survives even a power-cycle :-/
    [suggested by several polite people]

    Yes, this is definitely something I should do.
    I've seen one more occurrence of the problem,
    but didn't have a chance to test this - I probably inadvertently
    reset the UART and the problem went away before I tried a different
    terminal.
    I have Putty and CRT on my USB flash drive just in case.
    I know :-(
    It's a matter of not biting the hand that feeds you.
    After some early rookie experience in that way, I prefer to be
    cautious.
    After all, I'm not sure exactly where the problem is, and this
    decription
    could cast a shade on innocent hardware, which otherwise looks pretty
    good...
    right, tried that - made a file containing a megabyte of some
    character with the MSb=log.1,
    and the problem was still there. Back to 7bit characters, and still it
    was there.
    Ha, I haven't checked for IRDA. Makes me wonder if the IRDA modes
    possibly
    could use this kind of hardware feature. The UART's can work in some
    IRDA modes,
    but the hardware doesn't have a physical IR port, and as far as I
    know, the IR modes
    are not even offered in the BIOS setup for the COM ports. I can only
    hope they're
    properly disabled. Anyway I believe that Windows don't detect an IR
    port.
    Have to check to make sure - thanks for that note.
    Still it makes me wonder. If the UART device was seized by some IRDA
    service,
    the production app running on the PC would fail to open it as a COM
    port.
    Sounds weird...
    I was wondering if some Windows-internal ISA PnP stuff or generic
    SuperIO
    support could be messing with the SuperIO chip's config registers at
    random.
    Those are a special bank of registers with multiplexed access,
    separate from
    the classic ISA UART footprints in IO space.
    Perhaps more likely the BIOS could have some bugs in that direction
    (ACPI/SMI
    handling and whatnot). Wish I had an LPC bus probe :)
    No. It works just fine most of the time.
    The peers are set allright against each other.
    Seen such misconfigurations before. This is different.
    Most important it garbles every *other* character in a string.
    Thanks for the suggestion though :)
    It's a cut of XP Embedded based on XP SP3. I've checked the versions
    of
    serial.sys and serenum.sys - they're the same as on my laptop,
    with Windows Update active.
    no they're not :) You _don't_ use flow control on RS485.
    Anyway a flow control misconfiguration behaves very different.
    You don't get anything transmitted (the write() syscall blocks),
    or you get whole characters lost at the other end (FIFO overruns).
    [chuckle] don't tell me about well-meaning engineers :)
    What the Taiwanese HW designers can come up with...
    Such as auto-loopback relays on Ethernet ports, in hardware
    intended for firewall applications... or they use Linux in an embedded
    box,
    are soo very secretive about it (except that you can see the version
    strings
    in the firmware images available for download), and they can't seem to
    get *default routing* right, and they have bugs in serial port
    handling
    on a great 16C950 UART (those are two areas where Linux has been
    okay for decades now, in my experience).
    the production OS is XPe, derived from XP SP3 (Czech locale I
    believe).

    The Linux that I'm using for tests, is an in-house mini-distro
    assembled by hand
    (scripted file copy) based on Fedora 5, with a 2.6.28.6 kernel
    (custom config). Booting via PXE and working just fine with my
    tools :)

    ----

    The lasting problem is that I keep getting pieces of the suspected
    hardware
    RMA'ed due to irrelevant problems (software misconfigurations, RS485
    wiring sins),
    and only now and then an odd piece happens to show the "flipping bit"
    symptom.
    Based on the set of observations so far, I can't even tell if the
    "flipping bit syndrome in Hyperterminal" was to blame for a particular
    production
    malfunction, or if it was just the other problem I found on site :)
    This is something I have to sort through myselfs.

    The publishable results of this ongoing customer case so far have
    been:

    1) a snippet of Mingw C++ code that attempts to mimick the
    "hyperterminal reset"
    (I straced hyperterminal to see what it does at the level of
    syscalls,
    and then I tried to reproduce that at the level of DLL function
    calls :)
    http://www.fccps.cz/download/adv/frr/COMRESET.zip
    I've hooked it up into the startup sequence of the last culprit
    just in case.

    2) a brain-dump of collected thoughts on RS485 grounding and
    termination.
    My head tends to relax if I write things down an publish for the
    benefit of others
    (provided that they can read through the mess :)
    http://www.fccps.cz/download/adv/frr/RS485/RS485.html

    The important point is that I'm still having fun :)

    Thanks for your help everybody, take care

    Frank Rysanek
     
    Frantisek.Rysanek, Jan 26, 2010
    #10
  11. I had similar issues when I was writing a user interface for a machine
    that had a windows pc interfacing with a PLC. We were using win98 if
    I remember correctly. I think the uart and windows had an issue
    because of the way windows would multitask. Windows would lock the
    application just long enough to put it out of sync with the uart, and
    cause garbled data, or a frozen application. I think there were also
    problems with keeping other applications from accessing the port, even
    though it should have been locked. At first, I would have to restart
    windows when the app crashed. But then I rewrote the app, so I could
    just re-initiate communications when it failed. I have a binder
    around here somewhere, with our notes and data sheets and the like. I
    don't know if they'll apply to you, but I'll post a better explanation
    if I can find them.

    I must be getting old when I remember fixing something, but don't
    remember how the hell I did it. Then tell a long drawn out story with
    no useful information whatsoever.



    -Lazers
     
    Sansui Samari, Jan 28, 2010
    #11
  12. You might try posting here.
    http://omgili.com/probleme-rs232-sp3-windows-xp
     
    Sansui Samari, Jan 28, 2010
    #12
  13. And maybe searching through here...
    http://www.arcelect.com/technica.htm
     
    Sansui Samari, Jan 28, 2010
    #13
  14. Frantisek.Rysanek

    G B Guest

    Frank,

    I saw a similar problem but it was running at approx 1Mb/s. I had problems
    with clock skew in the devices. The clock in the send device was not the
    same as the receive device. Hence the clocks would slowly drift over time.
    I usually destroyed the 8th bit. Sometimes it would be just fine. I saw
    the part about every other character which made me think if there is no
    character pacing, it might take a full character time to resync. Just an
    idea..

    -Greg
    p.s. I liked your ground loop discussion and graphs. Good info!

    Dear gentlemen,

    many thanks to all of you who responded :)

    I'm delighted to see that the USENET is still alive, and that it's
    sometimes the same
    people responding to me, who used to respond ~10 years ago...

    And, maybe special thanks to Ian Malcolm - his bug description is the
    closest to mine :)
    If I exit Hyperterminal and start it again, the problem is typically
    still there.
    It survives even a power-cycle :-/
    [suggested by several polite people]

    Yes, this is definitely something I should do.
    I've seen one more occurrence of the problem,
    but didn't have a chance to test this - I probably inadvertently
    reset the UART and the problem went away before I tried a different
    terminal.
    I have Putty and CRT on my USB flash drive just in case.
    I know :-(
    It's a matter of not biting the hand that feeds you.
    After some early rookie experience in that way, I prefer to be
    cautious.
    After all, I'm not sure exactly where the problem is, and this
    decription
    could cast a shade on innocent hardware, which otherwise looks pretty
    good...
    right, tried that - made a file containing a megabyte of some
    character with the MSb=log.1,
    and the problem was still there. Back to 7bit characters, and still it
    was there.
    Ha, I haven't checked for IRDA. Makes me wonder if the IRDA modes
    possibly
    could use this kind of hardware feature. The UART's can work in some
    IRDA modes,
    but the hardware doesn't have a physical IR port, and as far as I
    know, the IR modes
    are not even offered in the BIOS setup for the COM ports. I can only
    hope they're
    properly disabled. Anyway I believe that Windows don't detect an IR
    port.
    Have to check to make sure - thanks for that note.
    Still it makes me wonder. If the UART device was seized by some IRDA
    service,
    the production app running on the PC would fail to open it as a COM
    port.
    Sounds weird...
    I was wondering if some Windows-internal ISA PnP stuff or generic
    SuperIO
    support could be messing with the SuperIO chip's config registers at
    random.
    Those are a special bank of registers with multiplexed access,
    separate from
    the classic ISA UART footprints in IO space.
    Perhaps more likely the BIOS could have some bugs in that direction
    (ACPI/SMI
    handling and whatnot). Wish I had an LPC bus probe :)
    No. It works just fine most of the time.
    The peers are set allright against each other.
    Seen such misconfigurations before. This is different.
    Most important it garbles every *other* character in a string.
    Thanks for the suggestion though :)
    It's a cut of XP Embedded based on XP SP3. I've checked the versions
    of
    serial.sys and serenum.sys - they're the same as on my laptop,
    with Windows Update active.
    no they're not :) You _don't_ use flow control on RS485.
    Anyway a flow control misconfiguration behaves very different.
    You don't get anything transmitted (the write() syscall blocks),
    or you get whole characters lost at the other end (FIFO overruns).
    [chuckle] don't tell me about well-meaning engineers :)
    What the Taiwanese HW designers can come up with...
    Such as auto-loopback relays on Ethernet ports, in hardware
    intended for firewall applications... or they use Linux in an embedded
    box,
    are soo very secretive about it (except that you can see the version
    strings
    in the firmware images available for download), and they can't seem to
    get *default routing* right, and they have bugs in serial port
    handling
    on a great 16C950 UART (those are two areas where Linux has been
    okay for decades now, in my experience).
    the production OS is XPe, derived from XP SP3 (Czech locale I
    believe).

    The Linux that I'm using for tests, is an in-house mini-distro
    assembled by hand
    (scripted file copy) based on Fedora 5, with a 2.6.28.6 kernel
    (custom config). Booting via PXE and working just fine with my
    tools :)

    ----

    The lasting problem is that I keep getting pieces of the suspected
    hardware
    RMA'ed due to irrelevant problems (software misconfigurations, RS485
    wiring sins),
    and only now and then an odd piece happens to show the "flipping bit"
    symptom.
    Based on the set of observations so far, I can't even tell if the
    "flipping bit syndrome in Hyperterminal" was to blame for a particular
    production
    malfunction, or if it was just the other problem I found on site :)
    This is something I have to sort through myselfs.

    The publishable results of this ongoing customer case so far have
    been:

    1) a snippet of Mingw C++ code that attempts to mimick the
    "hyperterminal reset"
    (I straced hyperterminal to see what it does at the level of
    syscalls,
    and then I tried to reproduce that at the level of DLL function
    calls :)
    http://www.fccps.cz/download/adv/frr/COMRESET.zip
    I've hooked it up into the startup sequence of the last culprit
    just in case.

    2) a brain-dump of collected thoughts on RS485 grounding and
    termination.
    My head tends to relax if I write things down an publish for the
    benefit of others
    (provided that they can read through the mess :)
    http://www.fccps.cz/download/adv/frr/RS485/RS485.html

    The important point is that I'm still having fun :)

    Thanks for your help everybody, take care

    Frank Rysanek
     
    G B, Jan 30, 2010
    #14
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.