1. This forum section is a read-only archive which contains old newsgroup posts. If you wish to post a query, please do so in one of our main forum sections (here). This way you will get a faster, better response from the members on Motherboard Point.

Special FAT filesystem questions [Was: Multiple files with same filename on FAT]

Discussion in 'Embedded' started by Vicne, Nov 6, 2007.

  1. Vicne

    Vicne Guest

    Hi,

    I'm still in the process of understanding the special FS my PVR uses
    (see "Multiple files with same filename on FAT" thread a few days
    ago).
    I read the spec of the MBR and BootSector / BPB, and manually walked
    through these parts.
    I can see that the first (and only) partition is declared the standard
    way as a 16-bit FAT (06h) in the MBR.
    Although, the boot sector of this partition clearly corresponds to a
    FAT32 (with a few weird values in reserved fields, but well). The last
    field (filesystem type) even reads "FAT32 ".

    I have a few questions though regarding the FAT itself (the actual
    linked list), that I'd like to submit to specialists (that's
    you :)) :
    - Can anybody explain me how to determine where the FAT starts on the
    disk ?
    - Does anybody have an idea what the following structure is :
    The first non-zero sector I find after the BS is at BE00h (offset in
    bytes from start of disk) and looks like some kind of linked list on
    24 (!) bits, stuffed with FF as fourth byte, like this :
    FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    ....
    (and there's a copy of it starting at 31200h)
    I guess it's not the FAT yet, right ?
    A few bytes after the end of the that copy (at 56800h precisely), what
    I think is the actual FAT begins, like this :
    81 00 00 00 82 00 00 00 83 00 00 00 84 00 00 00
    85 00 00 00 86 00 00 00 87 00 00 00 88 00 00 00
    89 00 00 00 8A 00 00 00 8B 00 00 00 8C 00 00 00
    ....
    (and there's a copy of this one starting at 12F6200h)

    Any hint or information is welcome, of course


    Vicne
     
    Vicne, Nov 6, 2007
    #1
    1. Advertisements

  2. Vicne

    Ted Davis Guest

    The discrepency between the file system identified by Ghost and the one
    identified by XP may be significant. It may even be a clue that the real
    file system is the same as the one used on the DVD, or at least similar to
    it (Joliet, perhaps)- that *would* simplify data transfers. If that just
    happened to be the case, you could be looking at an ISO 9660, Level 3, or
    UDF (ISO/IEC 13346) file system (with or without a bridge to ISO). The
    Wikipedia entries for those file systems are informative:
    <http://en.wikipedia.org/wiki/ISO_9660> and
    <http://en.wikipedia.org/wiki/Universal_Disk_Format>. Follow the link to
    ECMA-167
    (<http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-167.pdf>)
    for even more depth.
     
    Ted Davis, Nov 7, 2007
    #2
    1. Advertisements

  3. This isn't very relevant to the FAT on the partition, especially if
    that's the only partition out there. Don't take this discrepancy too
    close to heart. :)
    I would strongly recommend getting a copy of the MS FAT documentation
    from their web site and read through it:
    http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    It answers all the questions. Well, maybe not all, but the vast
    majority of them including this one. Then, in my code which you saw,
    there's a bunch of functions at the very beginning of FAT.c precisely
    for this sort of purpose: GetFirstFATSectorLBA(),
    GetFirstDataSectorLBA(), GetRootDirFirstSectorLBA() and the like.
    If you read the doc, you'll know it's effectively FAT28, look for 28
    in the doc. :)
    Seems like I have a minor bug in my code: I don't zero the 4 top bits
    in GetFATEntry() (or its callers) and I zero them out in
    SetFATEntry(). As per the doc both things aren't quite right. So, on
    your HDD my code will have problems because of this (and the wrong
    partition type, to begin with).
    Could be a copy. Check the bootsector for the number of FAT copies.
    HTH,
    Alex
     
    Alexei A. Frounze, Nov 7, 2007
    #3
  4. Vicne

    Vicne Guest

    Thanks for your reply.
    Nevertheless, preliminary analysis make me think this disk has a "99%"
    compatible FAT32 format. First mount attempts show that XP can mount
    it, navigate directories and read most of the files OK. The "weird"
    symptoms are random-looking or absent file timestamps, multiple
    entries with the same name and disk-ruining chkdsk in case you let it
    run :)

    Thanks for the suggestion anyway.

    Vicne
     
    Vicne, Nov 7, 2007
    #4
  5. Vicne

    Vicne Guest

    Noted :)

    Oh yes indeed. That's much more complete than the few pages I worked
    with. Thanks for the pointer.

    I quickly browsed the doc and I swear I'll read it thoroughly :), but
    reading your answer, it strikes me that the spec speaks about 28
    "address bits" indeed, while I observe 24 a full FF byte after 3
    variable ones...
    Oh, so "I" found a bug without even compiling the code ? :)
    It says 2.

    But what I find strange is that I have somthing like :
    "FAT24" (1st copy)
    "FAT24" (2nd copy)
    "FAT32" (1st copy)
    "FAT32" (2nd copy)

    The "FAT24" is only about 300 sectors long while the "FAT32" is about
    38000 sectors long.


    You see, I'm wondering if there isn't a kind of proprietary structure
    that would replace or complement the FAT in case of files > 4Gb, that
    would create dummy directory entries (with duplicate filenames) in the
    directory and would consist of a "chain of files" or something, stored
    before the FAT.

    On the other hand, I'm just a newbie regarding this as you know, so
    maybe I just can't see the obvious... I'll definitely read the doc and
    the relevant code you pointed me to, to see where exactly the boot
    sector says the FAT should begin.
    Hopefully, I'll start to write some code next week-end to test things
    out.

    Thanks a lot for your time and help. It's much appreciated.


    Vicne

    PS: BTW, Alex, do you have a recommendation regarding the OS for
    compiling and running your code ?
    I first planned to use FreeDOS but it seems it messed with my PVR
    clone disk (FreeDOS now appears in the first copy of the boot
    sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with
    it and the copy was truncated, while both XP and Linux performed the
    full copy without any problem.
    Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    floppy, "fdisk/format/sys c:" on the destination HD and copy over
    utils from the oldmsdos directory of Win98 CD.
    What do you think ? V.
     
    Vicne, Nov 7, 2007
    #5
  6. *I* found because I looked at the doc and code. You only guided me to
    do so. :)
    I don't know. Poke around. See if those things make sense, describe
    some data structure.
    Could be, although if you're certain with the previous discoveries
    that big files are just equally named 4G pieces, then it would be odd
    to have a different accounting for those 2nd and further parts. Then
    again, remembering the ext2fs-like FS, maybe not. It was something
    like:
    1st block/cluster/whatever of every file:
    - file data
    - ptr to block of N ptrs to next N data blocks
    - ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks
    - ptr to ... next N*N*N data blocks
    Compiling: DOS or 32-bit Windows (9x, 2k, xp).
    Running: DOS or bare machine with BIOS only.

    Read the documentation to see what tools you need and how things are
    compiled.
    That environment should be safe so long as you don't write anything to
    the camera's disk with DOS. :)
    Actually, you could greatly simplify your life by simply taking the
    raw image of that drive and tweaking the test code in image.c to the
    drive's geometry. That way you can use the test code to "mount" the
    image and work with it. The test code is portable C code (no ASM, no
    BIOS calls to do disk I/O) which you can easily tweak and debug under
    any OS you want. See imgcpy.c. It was written to copy files to/from a
    disk image file, almost what you need (you only need to figure out the
    details of the FS and tweak FAT.c to work with it and duplicated
    names). Just get fattest and imgcpy compiled and working as-is and
    then use your image instead of the test one and start digging,
    tweaking, etc.

    Alex
     
    Alexei A. Frounze, Nov 7, 2007
    #6
  7. - Can anybody explain me how to determine where the FAT starts on the
    The ReservedSectors BPB field is the number of sectors from volume start to the
    first FAT. It is usually 1 or sometimes 2 for FAT32 (FAT32 has an FsInfo sector
    which is not used by Windows NT's FASTFAT).
     
    Maxim S. Shatskih, Nov 7, 2007
    #7
  8. Vicne

    Ted Davis Guest

    All of which are indications that it is *not* FAT32.
     
    Ted Davis, Nov 7, 2007
    #8
  9. Although, the boot sector of this partition clearly corresponds to a
    Can you provide us with the exact Extended BPB values?
     
    Maxim S. Shatskih, Nov 7, 2007
    #9
  10. Vicne

    Matt Guest

    My guess would be that there might be a short FAT24 partition for the
    DVR to use for code updates, local storage etc. whilst the FAT32 is used
    for the actual finished MPEGS. The FAT24 might be hidden from general
    view in the same way that many computers have a hidden partition for
    diagnostics code.

    Matt


    Matt
     
    Matt, Nov 8, 2007
    #10
  11. Vicne

    Matt Guest

    Just a thought, but could the screwed up dates/times etc. be because the
    directory entry has used these fields to store the addresses or offsets
    to the next file chain, or next directory entry?

    Matt
     
    Matt, Nov 8, 2007
    #11
  12. Vicne

    Vicne Guest

    That would be strange to say the least, but well, many things seem
    strange to me for the moment. I guess the more I'll work on it, the
    clearer it will get.
    It could be... Although, most of the timestamps carried the same weird
    value at first, but well, that's an interesting track to follow. I'll
    pay more attention to these fields.

    Thanks for sharing your thoughts,

    Vicne
     
    Vicne, Nov 8, 2007
    #12
  13. Vicne

    Vicne Guest

    That's the BPB_RsvdSecCnt field at offset 14 I guess ? I read bytes 74
    then 02, or 274h. That's a very high value compared to what you say,
    but it makes sense: the Volume starts at byte 7E00h, and 7E00h + 274h
    * 200h = 56600h, and that's the beginning of the FAT32.
    As I said, the structure I called "FAT24" is about 300d sectors long,
    and there are 2 copies of it, and 274h = 628d, so it matches.
    Sure. I zipped and uploaded some interesting parts on
    http://www.megaupload.com/?d=TN8FZUV5 (it's only about 30 kB but I
    didn't know where to upload it). The zip contains
    - the bs_bpb part that began at byte 7E00h on disk
    - the fat_start that began at byte BE00h on disk
    - the video_dir (that's not the root dir but the subdir that contains
    the MPEG files, including some duplicate ones).

    Please note that these captures are not from a "perfect" image, but
    from a copy already mounted a few times, so it could be that some
    fields were messed up by XP...

    Thanks for your help.

    Vicne
     
    Vicne, Nov 8, 2007
    #13
  14. Vicne

    Vicne Guest

    Sure. I was not claiming any credit :)
    It looks like a linked list, but I'm going to leave it alone for now
    and concentrate on what is standard (or seems to be so). If I get
    stuck, I'll come back to it.
    Yes, indeed...
    Wow, some kind of tree, then ?
    Well, when I'll be a FAT-expert, I'll investigate those. Seems to be
    cool :)
    I have DJGPP installed and by screening your doc, it seems sufficient
    to build FAT module and test suite, so I'll begin with that.
    No, of course. Well, unless by mistake, I mean :)
    You're right. I'll try to do that.
    I guess only NTFS is suited to both support a 160 Gb file and be
    recognized under XP, right ? So I need an imaging tool that runs in
    plain DOS and can a bit-by-bit image write to an NTFS drive. I'm only
    familiar with Ghost however, but it writes images in its own format if
    I'm not mistaken... What tool would you advise ?
    I see. Great !
    Yes. Sounds the best plan indeed.

    Many thanks for your help


    Vicne
     
    Vicne, Nov 8, 2007
    #14
  15. Vicne

    Vicne Guest

    Not *exactly*, indeed :), but they share 99% of their "DNA".

    On the other hand, I quickly checked CD-ROM -
    http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf
    -
    and DVD structure - http://www.ecma-international.org/publications/files/ECMA-TR/TR-071.pdf
    -
    and they are completely different. Extensions (Rock Ridge, Joliet, El
    Torito) don't bring them any closer to what I'm seeing. Don't
    hesistate to take a look at the files on http://www.megaupload.com/?d=TN8FZUV5

    But you're right in that it looks like some "extension" to FAT32...

    Thanks for sharing your opinion.

    Vicne
     
    Vicne, Nov 10, 2007
    #15
  16. This guy has blog on developing a modified FAT for DVR's. Most interesting
    is summary 2:

    "However, at the same time, the Filesystem team developed an improved
    algorithm which should greatly help this situation: The original FAT code we
    were working with included a single per-open-file 32-bit cached entry of the
    last cluster accessed within the file, so sequential reads/writes always hit
    this cache and didn't need to retraverse the FAT chain"

    http://blogs.msdn.com/medmedia/archive/2007/01/04/fat-filesystem-performance-issues.aspx


    Anyway, MS has produced a number of custom FAT FS, such as ExFAT (WinCE),
    FATX (Xbox), TFAT. exFAT includes TFAT. TFAT uses multiple copies of the
    FAT table which I think you mentioned somewhere...

    FATX here:
    http://www.xbox-linux.org/wiki/Differences_between_Xbox_FATX_and_MS-DOS_FAT

    TFAT here:
    http://msdn2.microsoft.com/en-us/library/aa915463.aspx

    exFAT here:
    http://msdn2.microsoft.com/en-us/library/aa914353.aspx


    Rod Pemberton
     
    Rod Pemberton, Nov 10, 2007
    #16
  17. Well, what could've been done is either having a copy of FAT with
    cluster chains in the reverse order or a cache containing intermediate
    cluster numbers in the chain. To me there's not much benefit of having
    2 or more identical copies of FAT. If sh!t happens, tools will have
    hard times figuring out which one to trust and it's possible that
    neither is correct or they're both correct and wrong but in different
    places. So, why not just use the 2nd copy for that reversed linked
    list?

    Alex
     
    Alexei A. Frounze, Nov 11, 2007
    #17
  18. Well, I just mentioned it because I thought the cache might've been the
    "FAT24" the OP was talking about... But, I think guy mentioned why they
    didn't do what you suggested just prior to that quote (too many links):

    "Seeking within very large files: When seeking within a file, we have to
    traverse the file's FAT chain to find the desired cluster. As the file grows
    toward the 4GB filesize limit, the number of links that need to be traversed
    gets very large (for 32k clusters, up to 2^17 entries). As mentioned above,
    we partially avoided this issue by not using the full 4GB filesize available
    to us."

    I definately agree that two FAT's can confuse tools _if_ there is no way to
    distinquish which data is valid. I'm sure the point of two FAT's is to
    prevent corruption somehow. But, you'd have to tell me how this is done for
    FAT since I don't know. Filesize? Length of the files' linked list or
    "file chain?"

    The only disk format I've really looked at, decades ago, is the CBM format
    used in the C64's (and reviewed recently...). I've skimmed FAT12. FAT12
    contains the file's linked list in the FAT's, as you know. The CBM format
    distributes the files' linked list across the disk. Each track has a
    pointer to the next track and sector in the chain. The final track has a
    "track value" of zero and "sector value" indicating the offset into the
    sector of the final byte in the file. FAT12 is nice in that the data is
    always 512 bytes, not a few bytes less. Think about potential problems of
    reading 256 bytes from disk and then writing 254 bytes to memory instead of
    256 bytes into memory... The CBM is nice in that the link list is very
    simple, scales easily, is only limited by the size of the BAM (Block
    Availability Map) - a bitmap, and doesn't require increasing sector sizes
    for larger storage capacities, etc. The problem, as I remember it, with the
    CBM formats was when the BAM became corrupt for unknown reasons. You had to
    read in all the sectors and trace all the links to find the start of files
    and recreate the BAM. I'm not sure if the FAT's can be properly recreated,
    if corrupt, since it contains both the track and sector pointers of the
    files' linked list and the sector allocation map. Can it? For the CBM
    format, if the allocation map (BAM) was corrupt, the files' linked list was
    still valid. However, a problem arose when recreating the BAM, both an
    active file and numerous deleted files could point to the same track. If
    you undeleted the "files", you ended up with cross-linked files without any
    way to tell which file chain was good. The solution would've been simple,
    when a file is deleted, clear the track and sector pointers on deleted
    tracks. I'm sure they didn't do that at the time because it would've been
    slow to delete all at once. Today, of course, the FS should wipe the data
    anyway... or at least mark it for wiping later and delete it slowly and
    incrementally.


    Rod Pemberton
     
    Rod Pemberton, Nov 11, 2007
    #18
  19. Vicne

    Vicne Guest

    Hi,

    Thanks for these pointers. Very interesting reading indeed. I'm still
    hoping too this "FAT24" is only there for performance reasons and not
    absolutely needed to read and extract files.

    Kind regards,

    Vicne
     
    Vicne, Nov 11, 2007
    #19
  20. Vicne

    Vicne Guest

    I made progress in that direction yesterday :
    - built imginit and imgcpy,
    - played with them to create a dummy image and simulate the directory
    structure on my PVR (\VIDEO folder with a few files in it)
    - created a new 'imgdir' utility based on 'imgcpy', listing entries in
    a directory in both hex and "interpreted" form
    - replaced the dummy image by an image of the first sectors of the
    actual drive and dumped the entries in the \VIDEO directory

    All of this is working but I'm having difficulties determining what
    data was on the original copy and what was changed by mounting and
    accessing files under various OSes... Moreover, I only exported the
    first 35 Mb of the HD (MBR, BS/BPB, FAT, root dir, VIDEO subdir and
    the few first sectors of actual files) so no actual file extraction
    test can be performed...

    So working with a fresh image is the next step but I'm still stuck
    with a detail: how can I build a bit-by-bit image of a 160 Gb hard-
    drive without mounting it first under Windows...

    Anyway, thanks for providing the FAT toolkit, Alex, it's really great.


    Vicne
     
    Vicne, Nov 11, 2007
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.