1. This forum section is a read-only archive which contains old newsgroup posts. If you wish to post a query, please do so in one of our main forum sections (here). This way you will get a faster, better response from the members on Motherboard Point.

Special FAT filesystem questions [Was: Multiple files with same filename on FAT]

Discussion in 'Embedded' started by Vicne, Nov 6, 2007.

  1. Vicne

    Vicne Guest

    Hi,

    I'm still in the process of understanding the special FS my PVR uses
    (see "Multiple files with same filename on FAT" thread a few days
    ago).
    I read the spec of the MBR and BootSector / BPB, and manually walked
    through these parts.
    I can see that the first (and only) partition is declared the standard
    way as a 16-bit FAT (06h) in the MBR.
    Although, the boot sector of this partition clearly corresponds to a
    FAT32 (with a few weird values in reserved fields, but well). The last
    field (filesystem type) even reads "FAT32 ".

    I have a few questions though regarding the FAT itself (the actual
    linked list), that I'd like to submit to specialists (that's
    you :)) :
    - Can anybody explain me how to determine where the FAT starts on the
    disk ?
    - Does anybody have an idea what the following structure is :
    The first non-zero sector I find after the BS is at BE00h (offset in
    bytes from start of disk) and looks like some kind of linked list on
    24 (!) bits, stuffed with FF as fourth byte, like this :
    FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    ....
    (and there's a copy of it starting at 31200h)
    I guess it's not the FAT yet, right ?
    A few bytes after the end of the that copy (at 56800h precisely), what
    I think is the actual FAT begins, like this :
    81 00 00 00 82 00 00 00 83 00 00 00 84 00 00 00
    85 00 00 00 86 00 00 00 87 00 00 00 88 00 00 00
    89 00 00 00 8A 00 00 00 8B 00 00 00 8C 00 00 00
    ....
    (and there's a copy of this one starting at 12F6200h)

    Any hint or information is welcome, of course


    Vicne
    Vicne, Nov 6, 2007
    #1
    1. Advertising

  2. Vicne

    Ted Davis Guest

    On Tue, 06 Nov 2007 15:02:28 -0800, Vicne wrote:

    > Hi,
    >
    > I'm still in the process of understanding the special FS my PVR uses (see
    > "Multiple files with same filename on FAT" thread a few days ago).
    > I read the spec of the MBR and BootSector / BPB, and manually walked
    > through these parts.
    > I can see that the first (and only) partition is declared the standard way
    > as a 16-bit FAT (06h) in the MBR. Although, the boot sector of this
    > partition clearly corresponds to a FAT32 (with a few weird values in
    > reserved fields, but well). The last field (filesystem type) even reads
    > "FAT32 ".


    The discrepency between the file system identified by Ghost and the one
    identified by XP may be significant. It may even be a clue that the real
    file system is the same as the one used on the DVD, or at least similar to
    it (Joliet, perhaps)- that *would* simplify data transfers. If that just
    happened to be the case, you could be looking at an ISO 9660, Level 3, or
    UDF (ISO/IEC 13346) file system (with or without a bridge to ISO). The
    Wikipedia entries for those file systems are informative:
    <http://en.wikipedia.org/wiki/ISO_9660> and
    <http://en.wikipedia.org/wiki/Universal_Disk_Format>. Follow the link to
    ECMA-167
    (<http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-167.pdf>)
    for even more depth.

    --
    T.E.D. ()
    Ted Davis, Nov 7, 2007
    #2
    1. Advertising

  3. On Nov 6, 3:02 pm, Vicne <> wrote:
    > Hi,
    >
    > I'm still in the process of understanding the special FS my PVR uses
    > (see "Multiple files with same filename on FAT" thread a few days
    > ago).
    > I read the spec of the MBR and BootSector / BPB, and manually walked
    > through these parts.
    > I can see that the first (and only) partition is declared the standard
    > way as a 16-bit FAT (06h) in the MBR.


    This isn't very relevant to the FAT on the partition, especially if
    that's the only partition out there. Don't take this discrepancy too
    close to heart. :)

    > Although, the boot sector of this partition clearly corresponds to a
    > FAT32 (with a few weird values in reserved fields, but well). The last
    > field (filesystem type) even reads "FAT32 ".
    >
    > I have a few questions though regarding the FAT itself (the actual
    > linked list), that I'd like to submit to specialists (that's
    > you :)) :
    > - Can anybody explain me how to determine where the FAT starts on the
    > disk ?


    I would strongly recommend getting a copy of the MS FAT documentation
    from their web site and read through it:
    http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    It answers all the questions. Well, maybe not all, but the vast
    majority of them including this one. Then, in my code which you saw,
    there's a bunch of functions at the very beginning of FAT.c precisely
    for this sort of purpose: GetFirstFATSectorLBA(),
    GetFirstDataSectorLBA(), GetRootDirFirstSectorLBA() and the like.

    > - Does anybody have an idea what the following structure is :
    > The first non-zero sector I find after the BS is at BE00h (offset in
    > bytes from start of disk) and looks like some kind of linked list on
    > 24 (!) bits, stuffed with FF as fourth byte, like this :
    > FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    > DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    > 0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    > 0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    > 11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    > 15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    > ...


    If you read the doc, you'll know it's effectively FAT28, look for 28
    in the doc. :)
    Seems like I have a minor bug in my code: I don't zero the 4 top bits
    in GetFATEntry() (or its callers) and I zero them out in
    SetFATEntry(). As per the doc both things aren't quite right. So, on
    your HDD my code will have problems because of this (and the wrong
    partition type, to begin with).

    > (and there's a copy of it starting at 31200h)
    > I guess it's not the FAT yet, right ?


    Could be a copy. Check the bootsector for the number of FAT copies.

    > A few bytes after the end of the that copy (at 56800h precisely), what
    > I think is the actual FAT begins, like this :
    > 81 00 00 00 82 00 00 00 83 00 00 00 84 00 00 00
    > 85 00 00 00 86 00 00 00 87 00 00 00 88 00 00 00
    > 89 00 00 00 8A 00 00 00 8B 00 00 00 8C 00 00 00
    > ...
    > (and there's a copy of this one starting at 12F6200h)
    >
    > Any hint or information is welcome, of course


    HTH,
    Alex
    Alexei A. Frounze, Nov 7, 2007
    #3
  4. Vicne

    Vicne Guest

    On Nov 7, 2:45 am, Ted Davis <> wrote:

    > The discrepency between the file system identified by Ghost and the one
    > identified by XP may be significant. It may even be a clue that the real
    > file system is the same as the one used on the DVD, or at least similar to
    > it (Joliet, perhaps)- that *would* simplify data transfers.


    Thanks for your reply.
    Nevertheless, preliminary analysis make me think this disk has a "99%"
    compatible FAT32 format. First mount attempts show that XP can mount
    it, navigate directories and read most of the files OK. The "weird"
    symptoms are random-looking or absent file timestamps, multiple
    entries with the same name and disk-ruining chkdsk in case you let it
    run :)

    Thanks for the suggestion anyway.

    Vicne
    Vicne, Nov 7, 2007
    #4
  5. Vicne

    Vicne Guest

    On Nov 7, 7:07 am, "Alexei A. Frounze" <> wrote:
    > This isn't very relevant to the FAT on the partition, especially if
    > that's the only partition out there. Don't take this discrepancy too
    > close to heart. :)


    Noted :)


    > I would strongly recommend getting a copy of the MS FAT documentation
    > from their web site and read through it:http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    > It answers all the questions. Well, maybe not all, but the vast
    > majority of them including this one.


    Oh yes indeed. That's much more complete than the few pages I worked
    with. Thanks for the pointer.


    > > FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    > > DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    > > 0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    > > 0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    > > 11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    > > 15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    > > ...

    >
    > If you read the doc, you'll know it's effectively FAT28, look for 28
    > in the doc. :)


    I quickly browsed the doc and I swear I'll read it thoroughly :), but
    reading your answer, it strikes me that the spec speaks about 28
    "address bits" indeed, while I observe 24 a full FF byte after 3
    variable ones...

    > Seems like I have a minor bug in my code: I don't zero the 4 top bits
    > in GetFATEntry() (or its callers) and I zero them out in
    > SetFATEntry(). As per the doc both things aren't quite right. So, on
    > your HDD my code will have problems because of this (and the wrong
    > partition type, to begin with).


    Oh, so "I" found a bug without even compiling the code ? :)

    > > (and there's a copy of it starting at 31200h)
    > > I guess it's not the FAT yet, right ?

    >
    > Could be a copy. Check the bootsector for the number of FAT copies


    It says 2.

    But what I find strange is that I have somthing like :
    "FAT24" (1st copy)
    "FAT24" (2nd copy)
    "FAT32" (1st copy)
    "FAT32" (2nd copy)

    The "FAT24" is only about 300 sectors long while the "FAT32" is about
    38000 sectors long.


    You see, I'm wondering if there isn't a kind of proprietary structure
    that would replace or complement the FAT in case of files > 4Gb, that
    would create dummy directory entries (with duplicate filenames) in the
    directory and would consist of a "chain of files" or something, stored
    before the FAT.

    On the other hand, I'm just a newbie regarding this as you know, so
    maybe I just can't see the obvious... I'll definitely read the doc and
    the relevant code you pointed me to, to see where exactly the boot
    sector says the FAT should begin.
    Hopefully, I'll start to write some code next week-end to test things
    out.

    Thanks a lot for your time and help. It's much appreciated.


    Vicne

    PS: BTW, Alex, do you have a recommendation regarding the OS for
    compiling and running your code ?
    I first planned to use FreeDOS but it seems it messed with my PVR
    clone disk (FreeDOS now appears in the first copy of the boot
    sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with
    it and the copy was truncated, while both XP and Linux performed the
    full copy without any problem.
    Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    floppy, "fdisk/format/sys c:" on the destination HD and copy over
    utils from the oldmsdos directory of Win98 CD.
    What do you think ? V.
    Vicne, Nov 7, 2007
    #5
  6. On Nov 7, 1:16 am, Vicne <> wrote:
    > On Nov 7, 7:07 am, "Alexei A. Frounze" <> wrote:
    >
    > > This isn't very relevant to the FAT on the partition, especially if
    > > that's the only partition out there. Don't take this discrepancy too
    > > close to heart. :)

    >
    > Noted :)
    >
    > > I would strongly recommend getting a copy of the MS FAT documentation
    > > from their web site and read through it:http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    > > It answers all the questions. Well, maybe not all, but the vast
    > > majority of them including this one.

    >
    > Oh yes indeed. That's much more complete than the few pages I worked
    > with. Thanks for the pointer.
    >
    > > > FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    > > > DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    > > > 0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    > > > 0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    > > > 11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    > > > 15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    > > > ...

    >
    > > If you read the doc, you'll know it's effectively FAT28, look for 28
    > > in the doc. :)

    >
    > I quickly browsed the doc and I swear I'll read it thoroughly :), but
    > reading your answer, it strikes me that the spec speaks about 28
    > "address bits" indeed, while I observe 24 a full FF byte after 3
    > variable ones...
    >
    > > Seems like I have a minor bug in my code: I don't zero the 4 top bits
    > > in GetFATEntry() (or its callers) and I zero them out in
    > > SetFATEntry(). As per the doc both things aren't quite right. So, on
    > > your HDD my code will have problems because of this (and the wrong
    > > partition type, to begin with).

    >
    > Oh, so "I" found a bug without even compiling the code ? :)


    *I* found because I looked at the doc and code. You only guided me to
    do so. :)

    > > > (and there's a copy of it starting at 31200h)
    > > > I guess it's not the FAT yet, right ?

    >
    > > Could be a copy. Check the bootsector for the number of FAT copies

    >
    > It says 2.
    >
    > But what I find strange is that I have somthing like :
    > "FAT24" (1st copy)
    > "FAT24" (2nd copy)
    > "FAT32" (1st copy)
    > "FAT32" (2nd copy)
    >
    > The "FAT24" is only about 300 sectors long while the "FAT32" is about
    > 38000 sectors long.


    I don't know. Poke around. See if those things make sense, describe
    some data structure.

    > You see, I'm wondering if there isn't a kind of proprietary structure
    > that would replace or complement the FAT in case of files > 4Gb, that
    > would create dummy directory entries (with duplicate filenames) in the
    > directory and would consist of a "chain of files" or something, stored
    > before the FAT.


    Could be, although if you're certain with the previous discoveries
    that big files are just equally named 4G pieces, then it would be odd
    to have a different accounting for those 2nd and further parts. Then
    again, remembering the ext2fs-like FS, maybe not. It was something
    like:
    1st block/cluster/whatever of every file:
    - file data
    - ptr to block of N ptrs to next N data blocks
    - ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks
    - ptr to ... next N*N*N data blocks

    > On the other hand, I'm just a newbie regarding this as you know, so
    > maybe I just can't see the obvious... I'll definitely read the doc and
    > the relevant code you pointed me to, to see where exactly the boot
    > sector says the FAT should begin.
    > Hopefully, I'll start to write some code next week-end to test things
    > out.
    >
    > Thanks a lot for your time and help. It's much appreciated.
    >
    > Vicne
    >
    > PS: BTW, Alex, do you have a recommendation regarding the OS for
    > compiling and running your code ?


    Compiling: DOS or 32-bit Windows (9x, 2k, xp).
    Running: DOS or bare machine with BIOS only.

    Read the documentation to see what tools you need and how things are
    compiled.

    > I first planned to use FreeDOS but it seems it messed with my PVR
    > clone disk (FreeDOS now appears in the first copy of the boot
    > sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with
    > it and the copy was truncated, while both XP and Linux performed the
    > full copy without any problem.
    > Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    > floppy, "fdisk/format/sys c:" on the destination HD and copy over
    > utils from the oldmsdos directory of Win98 CD.
    > What do you think ? V.


    That environment should be safe so long as you don't write anything to
    the camera's disk with DOS. :)
    Actually, you could greatly simplify your life by simply taking the
    raw image of that drive and tweaking the test code in image.c to the
    drive's geometry. That way you can use the test code to "mount" the
    image and work with it. The test code is portable C code (no ASM, no
    BIOS calls to do disk I/O) which you can easily tweak and debug under
    any OS you want. See imgcpy.c. It was written to copy files to/from a
    disk image file, almost what you need (you only need to figure out the
    details of the FS and tweak FAT.c to work with it and duplicated
    names). Just get fattest and imgcpy compiled and working as-is and
    then use your image instead of the test one and start digging,
    tweaking, etc.

    Alex
    Alexei A. Frounze, Nov 7, 2007
    #6
  7. > - Can anybody explain me how to determine where the FAT starts on the
    > disk ?


    The ReservedSectors BPB field is the number of sectors from volume start to the
    first FAT. It is usually 1 or sometimes 2 for FAT32 (FAT32 has an FsInfo sector
    which is not used by Windows NT's FASTFAT).

    --
    Maxim Shatskih, Windows DDK MVP
    StorageCraft Corporation

    http://www.storagecraft.com
    Maxim S. Shatskih, Nov 7, 2007
    #7
  8. Vicne

    Ted Davis Guest

    On Wed, 07 Nov 2007 00:26:29 -0800, Vicne wrote:

    > On Nov 7, 2:45 am, Ted Davis <> wrote:
    >
    >> The discrepency between the file system identified by Ghost and the one
    >> identified by XP may be significant. It may even be a clue that the real
    >> file system is the same as the one used on the DVD, or at least similar
    >> to it (Joliet, perhaps)- that *would* simplify data transfers.

    >
    > Thanks for your reply.
    > Nevertheless, preliminary analysis make me think this disk has a "99%"
    > compatible FAT32 format. First mount attempts show that XP can mount it,
    > navigate directories and read most of the files OK. The "weird" symptoms
    > are random-looking or absent file timestamps, multiple entries with the
    > same name and disk-ruining chkdsk in case you let it run :)
    >

    All of which are indications that it is *not* FAT32.

    --
    T.E.D. ()
    Ted Davis, Nov 7, 2007
    #8
  9. > Although, the boot sector of this partition clearly corresponds to a
    > FAT32 (with a few weird values in reserved fields, but well). The last
    > field (filesystem type) even reads "FAT32 ".


    Can you provide us with the exact Extended BPB values?

    --
    Maxim Shatskih, Windows DDK MVP
    StorageCraft Corporation

    http://www.storagecraft.com
    Maxim S. Shatskih, Nov 7, 2007
    #9
  10. Vicne

    Matt Guest

    Re: Special FAT filesystem questions [Was: Multiple files with samefilename on FAT]

    Alexei A. Frounze wrote:
    > On Nov 7, 1:16 am, Vicne <> wrote:
    >> On Nov 7, 7:07 am, "Alexei A. Frounze" <> wrote:
    >>
    >>> This isn't very relevant to the FAT on the partition, especially if
    >>> that's the only partition out there. Don't take this discrepancy too
    >>> close to heart. :)

    >> Noted :)
    >>
    >>> I would strongly recommend getting a copy of the MS FAT documentation
    >>> from their web site and read through it:http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    >>> It answers all the questions. Well, maybe not all, but the vast
    >>> majority of them including this one.

    >> Oh yes indeed. That's much more complete than the few pages I worked
    >> with. Thanks for the pointer.
    >>
    >>>> FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    >>>> DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    >>>> 0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    >>>> 0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    >>>> 11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    >>>> 15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    >>>> ...
    >>> If you read the doc, you'll know it's effectively FAT28, look for 28
    >>> in the doc. :)

    >> I quickly browsed the doc and I swear I'll read it thoroughly :), but
    >> reading your answer, it strikes me that the spec speaks about 28
    >> "address bits" indeed, while I observe 24 a full FF byte after 3
    >> variable ones...
    >>
    >>> Seems like I have a minor bug in my code: I don't zero the 4 top bits
    >>> in GetFATEntry() (or its callers) and I zero them out in
    >>> SetFATEntry(). As per the doc both things aren't quite right. So, on
    >>> your HDD my code will have problems because of this (and the wrong
    >>> partition type, to begin with).

    >> Oh, so "I" found a bug without even compiling the code ? :)

    >
    > *I* found because I looked at the doc and code. You only guided me to
    > do so. :)
    >
    >>>> (and there's a copy of it starting at 31200h)
    >>>> I guess it's not the FAT yet, right ?
    >>> Could be a copy. Check the bootsector for the number of FAT copies

    >> It says 2.
    >>
    >> But what I find strange is that I have somthing like :
    >> "FAT24" (1st copy)
    >> "FAT24" (2nd copy)
    >> "FAT32" (1st copy)
    >> "FAT32" (2nd copy)
    >>
    >> The "FAT24" is only about 300 sectors long while the "FAT32" is about
    >> 38000 sectors long.

    >
    > I don't know. Poke around. See if those things make sense, describe
    > some data structure.
    >
    >> You see, I'm wondering if there isn't a kind of proprietary structure
    >> that would replace or complement the FAT in case of files > 4Gb, that
    >> would create dummy directory entries (with duplicate filenames) in the
    >> directory and would consist of a "chain of files" or something, stored
    >> before the FAT.

    >
    > Could be, although if you're certain with the previous discoveries
    > that big files are just equally named 4G pieces, then it would be odd
    > to have a different accounting for those 2nd and further parts. Then
    > again, remembering the ext2fs-like FS, maybe not. It was something
    > like:
    > 1st block/cluster/whatever of every file:
    > - file data
    > - ptr to block of N ptrs to next N data blocks
    > - ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks
    > - ptr to ... next N*N*N data blocks
    >
    >> On the other hand, I'm just a newbie regarding this as you know, so
    >> maybe I just can't see the obvious... I'll definitely read the doc and
    >> the relevant code you pointed me to, to see where exactly the boot
    >> sector says the FAT should begin.
    >> Hopefully, I'll start to write some code next week-end to test things
    >> out.
    >>
    >> Thanks a lot for your time and help. It's much appreciated.
    >>
    >> Vicne
    >>
    >> PS: BTW, Alex, do you have a recommendation regarding the OS for
    >> compiling and running your code ?

    >
    > Compiling: DOS or 32-bit Windows (9x, 2k, xp).
    > Running: DOS or bare machine with BIOS only.
    >
    > Read the documentation to see what tools you need and how things are
    > compiled.
    >
    >> I first planned to use FreeDOS but it seems it messed with my PVR
    >> clone disk (FreeDOS now appears in the first copy of the boot
    >> sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with
    >> it and the copy was truncated, while both XP and Linux performed the
    >> full copy without any problem.
    >> Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    >> floppy, "fdisk/format/sys c:" on the destination HD and copy over
    >> utils from the oldmsdos directory of Win98 CD.
    >> What do you think ? V.

    >
    > That environment should be safe so long as you don't write anything to
    > the camera's disk with DOS. :)
    > Actually, you could greatly simplify your life by simply taking the
    > raw image of that drive and tweaking the test code in image.c to the
    > drive's geometry. That way you can use the test code to "mount" the
    > image and work with it. The test code is portable C code (no ASM, no
    > BIOS calls to do disk I/O) which you can easily tweak and debug under
    > any OS you want. See imgcpy.c. It was written to copy files to/from a
    > disk image file, almost what you need (you only need to figure out the
    > details of the FS and tweak FAT.c to work with it and duplicated
    > names). Just get fattest and imgcpy compiled and working as-is and
    > then use your image instead of the test one and start digging,
    > tweaking, etc.
    >
    > Alex
    >

    My guess would be that there might be a short FAT24 partition for the
    DVR to use for code updates, local storage etc. whilst the FAT32 is used
    for the actual finished MPEGS. The FAT24 might be hidden from general
    view in the same way that many computers have a hidden partition for
    diagnostics code.

    Matt


    Matt
    Matt, Nov 8, 2007
    #10
  11. Vicne

    Matt Guest

    Re: Special FAT filesystem questions [Was: Multiple files with samefilename on FAT]

    Alexei A. Frounze wrote:
    > On Nov 7, 1:16 am, Vicne <> wrote:
    >> On Nov 7, 7:07 am, "Alexei A. Frounze" <> wrote:
    >>
    >>> This isn't very relevant to the FAT on the partition, especially if
    >>> that's the only partition out there. Don't take this discrepancy too
    >>> close to heart. :)

    >> Noted :)
    >>
    >>> I would strongly recommend getting a copy of the MS FAT documentation
    >>> from their web site and read through it:http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
    >>> It answers all the questions. Well, maybe not all, but the vast
    >>> majority of them including this one.

    >> Oh yes indeed. That's much more complete than the few pages I worked
    >> with. Thanks for the pointer.
    >>
    >>>> FF FF 1F 09 FF FF 1F FF 04 00 00 FF FF FF 1F FF
    >>>> DF 07 00 FF FF FF 1F FF 07 00 00 FF 08 00 00 FF
    >>>> 0A 00 00 FF FF FF 1F FF 0B 00 00 FF 0C 00 00 FF
    >>>> 0E 00 00 FF FF FF 1F FF 0F 00 00 FF 10 00 00 FF
    >>>> 11 00 00 FF 12 00 00 FF 13 00 00 FF 14 00 00 FF
    >>>> 15 00 00 FF 16 00 00 FF 17 00 00 FF 18 00 00 FF
    >>>> ...
    >>> If you read the doc, you'll know it's effectively FAT28, look for 28
    >>> in the doc. :)

    >> I quickly browsed the doc and I swear I'll read it thoroughly :), but
    >> reading your answer, it strikes me that the spec speaks about 28
    >> "address bits" indeed, while I observe 24 a full FF byte after 3
    >> variable ones...
    >>
    >>> Seems like I have a minor bug in my code: I don't zero the 4 top bits
    >>> in GetFATEntry() (or its callers) and I zero them out in
    >>> SetFATEntry(). As per the doc both things aren't quite right. So, on
    >>> your HDD my code will have problems because of this (and the wrong
    >>> partition type, to begin with).

    >> Oh, so "I" found a bug without even compiling the code ? :)

    >
    > *I* found because I looked at the doc and code. You only guided me to
    > do so. :)
    >
    >>>> (and there's a copy of it starting at 31200h)
    >>>> I guess it's not the FAT yet, right ?
    >>> Could be a copy. Check the bootsector for the number of FAT copies

    >> It says 2.
    >>
    >> But what I find strange is that I have somthing like :
    >> "FAT24" (1st copy)
    >> "FAT24" (2nd copy)
    >> "FAT32" (1st copy)
    >> "FAT32" (2nd copy)
    >>
    >> The "FAT24" is only about 300 sectors long while the "FAT32" is about
    >> 38000 sectors long.

    >
    > I don't know. Poke around. See if those things make sense, describe
    > some data structure.
    >
    >> You see, I'm wondering if there isn't a kind of proprietary structure
    >> that would replace or complement the FAT in case of files > 4Gb, that
    >> would create dummy directory entries (with duplicate filenames) in the
    >> directory and would consist of a "chain of files" or something, stored
    >> before the FAT.

    >
    > Could be, although if you're certain with the previous discoveries
    > that big files are just equally named 4G pieces, then it would be odd
    > to have a different accounting for those 2nd and further parts. Then
    > again, remembering the ext2fs-like FS, maybe not. It was something
    > like:
    > 1st block/cluster/whatever of every file:
    > - file data
    > - ptr to block of N ptrs to next N data blocks
    > - ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks
    > - ptr to ... next N*N*N data blocks
    >
    >> On the other hand, I'm just a newbie regarding this as you know, so
    >> maybe I just can't see the obvious... I'll definitely read the doc and
    >> the relevant code you pointed me to, to see where exactly the boot
    >> sector says the FAT should begin.
    >> Hopefully, I'll start to write some code next week-end to test things
    >> out.
    >>
    >> Thanks a lot for your time and help. It's much appreciated.
    >>
    >> Vicne
    >>
    >> PS: BTW, Alex, do you have a recommendation regarding the OS for
    >> compiling and running your code ?

    >
    > Compiling: DOS or 32-bit Windows (9x, 2k, xp).
    > Running: DOS or bare machine with BIOS only.
    >
    > Read the documentation to see what tools you need and how things are
    > compiled.
    >
    >> I first planned to use FreeDOS but it seems it messed with my PVR
    >> clone disk (FreeDOS now appears in the first copy of the boot
    >> sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with
    >> it and the copy was truncated, while both XP and Linux performed the
    >> full copy without any problem.
    >> Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    >> floppy, "fdisk/format/sys c:" on the destination HD and copy over
    >> utils from the oldmsdos directory of Win98 CD.
    >> What do you think ? V.

    >
    > That environment should be safe so long as you don't write anything to
    > the camera's disk with DOS. :)
    > Actually, you could greatly simplify your life by simply taking the
    > raw image of that drive and tweaking the test code in image.c to the
    > drive's geometry. That way you can use the test code to "mount" the
    > image and work with it. The test code is portable C code (no ASM, no
    > BIOS calls to do disk I/O) which you can easily tweak and debug under
    > any OS you want. See imgcpy.c. It was written to copy files to/from a
    > disk image file, almost what you need (you only need to figure out the
    > details of the FS and tweak FAT.c to work with it and duplicated
    > names). Just get fattest and imgcpy compiled and working as-is and
    > then use your image instead of the test one and start digging,
    > tweaking, etc.
    >
    > Alex
    >


    Just a thought, but could the screwed up dates/times etc. be because the
    directory entry has used these fields to store the addresses or offsets
    to the next file chain, or next directory entry?

    Matt
    Matt, Nov 8, 2007
    #11
  12. Vicne

    Vicne Guest

    Matt <> wrote:
    > My guess would be that there might be a short FAT24 partition for the
    > DVR to use for code updates, local storage etc. whilst the FAT32 is used
    > for the actual finished MPEGS. The FAT24 might be hidden from general
    > view in the same way that many computers have a hidden partition for
    > diagnostics code.


    That would be strange to say the least, but well, many things seem
    strange to me for the moment. I guess the more I'll work on it, the
    clearer it will get.

    > Just a thought, but could the screwed up dates/times etc. be because the
    > directory entry has used these fields to store the addresses or offsets
    > to the next file chain, or next directory entry?


    It could be... Although, most of the timestamps carried the same weird
    value at first, but well, that's an interesting track to follow. I'll
    pay more attention to these fields.

    Thanks for sharing your thoughts,

    Vicne
    Vicne, Nov 8, 2007
    #12
  13. Vicne

    Vicne Guest

    On Nov 7, 11:42 am, "Maxim S. Shatskih" <>
    wrote:
    > > - Can anybody explain me how to determine where the FAT starts on the
    > > disk ?

    >
    > The ReservedSectors BPB field is the number of sectors from volume start to the
    > first FAT. It is usually 1 or sometimes 2 for FAT32 (FAT32 has an FsInfo sector
    > which is not used by Windows NT's FASTFAT).


    That's the BPB_RsvdSecCnt field at offset 14 I guess ? I read bytes 74
    then 02, or 274h. That's a very high value compared to what you say,
    but it makes sense: the Volume starts at byte 7E00h, and 7E00h + 274h
    * 200h = 56600h, and that's the beginning of the FAT32.
    As I said, the structure I called "FAT24" is about 300d sectors long,
    and there are 2 copies of it, and 274h = 628d, so it matches.

    > Can you provide us with the exact Extended BPB values?


    Sure. I zipped and uploaded some interesting parts on
    http://www.megaupload.com/?d=TN8FZUV5 (it's only about 30 kB but I
    didn't know where to upload it). The zip contains
    - the bs_bpb part that began at byte 7E00h on disk
    - the fat_start that began at byte BE00h on disk
    - the video_dir (that's not the root dir but the subdir that contains
    the MPEG files, including some duplicate ones).

    Please note that these captures are not from a "perfect" image, but
    from a copy already mounted a few times, so it could be that some
    fields were messed up by XP...

    Thanks for your help.

    Vicne
    Vicne, Nov 8, 2007
    #13
  14. Vicne

    Vicne Guest

    On Nov 7, 11:36 am, "Alexei A. Frounze" <> wrote:
    > > Oh, so "I" found a bug without even compiling the code ? :)

    >
    > *I* found because I looked at the doc and code. You only guided me to
    > do so. :)


    Sure. I was not claiming any credit :)

    > I don't know. Poke around. See if those things make sense, describe
    > some data structure.


    It looks like a linked list, but I'm going to leave it alone for now
    and concentrate on what is standard (or seems to be so). If I get
    stuck, I'll come back to it.

    > Could be, although if you're certain with the previous discoveries
    > that big files are just equally named 4G pieces, then it would be odd
    > to have a different accounting for those 2nd and further parts.


    Yes, indeed...

    > Then again, remembering the ext2fs-like FS, maybe not. It was something
    > like:
    > 1st block/cluster/whatever of every file:
    > - file data
    > - ptr to block of N ptrs to next N data blocks
    > - ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks
    > - ptr to ... next N*N*N data blocks


    Wow, some kind of tree, then ?
    Well, when I'll be a FAT-expert, I'll investigate those. Seems to be
    cool :)

    > Compiling: DOS or 32-bit Windows (9x, 2k, xp).
    > Running: DOS or bare machine with BIOS only.


    OK.

    > Read the documentation to see what tools you need and how things are
    > compiled.


    I have DJGPP installed and by screening your doc, it seems sufficient
    to build FAT module and test suite, so I'll begin with that.

    > > Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on
    > > floppy, "fdisk/format/sys c:" on the destination HD and copy over
    > > utils from the oldmsdos directory of Win98 CD.
    > > What do you think ? V.

    >
    > That environment should be safe so long as you don't write anything to
    > the camera's disk with DOS. :)


    No, of course. Well, unless by mistake, I mean :)

    > Actually, you could greatly simplify your life by simply taking the
    > raw image of that drive and tweaking the test code in image.c to the
    > drive's geometry.


    You're right. I'll try to do that.
    I guess only NTFS is suited to both support a 160 Gb file and be
    recognized under XP, right ? So I need an imaging tool that runs in
    plain DOS and can a bit-by-bit image write to an NTFS drive. I'm only
    familiar with Ghost however, but it writes images in its own format if
    I'm not mistaken... What tool would you advise ?

    > That way you can use the test code to "mount" the
    > image and work with it. The test code is portable C code (no ASM, no
    > BIOS calls to do disk I/O) which you can easily tweak and debug under
    > any OS you want.


    I see. Great !

    > See imgcpy.c. It was written to copy files to/from a
    > disk image file, almost what you need (you only need to figure out the
    > details of the FS and tweak FAT.c to work with it and duplicated
    > names). Just get fattest and imgcpy compiled and working as-is and
    > then use your image instead of the test one and start digging,
    > tweaking, etc.


    Yes. Sounds the best plan indeed.

    Many thanks for your help


    Vicne
    Vicne, Nov 8, 2007
    #14
  15. Vicne

    Vicne Guest

    On 7 nov, 14:48, Ted Davis <> wrote:
    > > Nevertheless, preliminary analysis make me think this disk has a "99%"
    > > compatible FAT32 format. First mount attempts show that XP can mount it,
    > > navigate directories and read most of the files OK. The "weird" symptoms
    > > are random-looking or absent file timestamps, multiple entries with the
    > > same name and disk-ruining chkdsk in case you let it run :)

    >
    > All of which are indications that it is *not* FAT32.


    Not *exactly*, indeed :), but they share 99% of their "DNA".

    On the other hand, I quickly checked CD-ROM -
    http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf
    -
    and DVD structure - http://www.ecma-international.org/publications/files/ECMA-TR/TR-071.pdf
    -
    and they are completely different. Extensions (Rock Ridge, Joliet, El
    Torito) don't bring them any closer to what I'm seeing. Don't
    hesistate to take a look at the files on http://www.megaupload.com/?d=TN8FZUV5

    But you're right in that it looks like some "extension" to FAT32...

    Thanks for sharing your opinion.

    Vicne
    Vicne, Nov 10, 2007
    #15
  16. "Vicne" <> wrote in message
    news:...
    > On 7 nov, 14:48, Ted Davis <> wrote:
    > > > Nevertheless, preliminary analysis make me think this disk has a "99%"
    > > > compatible FAT32 format. First mount attempts show that XP can mount

    it,
    > > > navigate directories and read most of the files OK. The "weird"

    symptoms
    > > > are random-looking or absent file timestamps, multiple entries with

    the
    > > > same name and disk-ruining chkdsk in case you let it run :)

    > >
    > > All of which are indications that it is *not* FAT32.

    >
    > Not *exactly*, indeed :), but they share 99% of their "DNA".
    >
    > On the other hand, I quickly checked

    <snip>
    >
    > But you're right in that it looks like some "extension" to FAT32...
    >


    This guy has blog on developing a modified FAT for DVR's. Most interesting
    is summary 2:

    "However, at the same time, the Filesystem team developed an improved
    algorithm which should greatly help this situation: The original FAT code we
    were working with included a single per-open-file 32-bit cached entry of the
    last cluster accessed within the file, so sequential reads/writes always hit
    this cache and didn't need to retraverse the FAT chain"

    http://blogs.msdn.com/medmedia/archive/2007/01/04/fat-filesystem-performance-issues.aspx


    Anyway, MS has produced a number of custom FAT FS, such as ExFAT (WinCE),
    FATX (Xbox), TFAT. exFAT includes TFAT. TFAT uses multiple copies of the
    FAT table which I think you mentioned somewhere...

    FATX here:
    http://www.xbox-linux.org/wiki/Differences_between_Xbox_FATX_and_MS-DOS_FAT

    TFAT here:
    http://msdn2.microsoft.com/en-us/library/aa915463.aspx

    exFAT here:
    http://msdn2.microsoft.com/en-us/library/aa914353.aspx


    Rod Pemberton
    Rod Pemberton, Nov 10, 2007
    #16
  17. On Nov 10, 12:38 pm, "Rod Pemberton" <>
    wrote:
    > "Vicne" <> wrote in message
    >
    > news:...
    >
    >
    >
    > > On 7 nov, 14:48, Ted Davis <> wrote:
    > > > > Nevertheless, preliminary analysis make me think this disk has a "99%"
    > > > > compatible FAT32 format. First mount attempts show that XP can mount

    > it,
    > > > > navigate directories and read most of the files OK. The "weird"

    > symptoms
    > > > > are random-looking or absent file timestamps, multiple entries with

    > the
    > > > > same name and disk-ruining chkdsk in case you let it run :)

    >
    > > > All of which are indications that it is *not* FAT32.

    >
    > > Not *exactly*, indeed :), but they share 99% of their "DNA".

    >
    > > On the other hand, I quickly checked

    > <snip>
    >
    > > But you're right in that it looks like some "extension" to FAT32...

    >
    > This guy has blog on developing a modified FAT for DVR's. Most interesting
    > is summary 2:
    >
    > "However, at the same time, the Filesystem team developed an improved
    > algorithm which should greatly help this situation: The original FAT code we
    > were working with included a single per-open-file 32-bit cached entry of the
    > last cluster accessed within the file, so sequential reads/writes always hit
    > this cache and didn't need to retraverse the FAT chain"
    >
    > http://blogs.msdn.com/medmedia/archive/2007/01/04/fat-filesystem-perf...


    Well, what could've been done is either having a copy of FAT with
    cluster chains in the reverse order or a cache containing intermediate
    cluster numbers in the chain. To me there's not much benefit of having
    2 or more identical copies of FAT. If sh!t happens, tools will have
    hard times figuring out which one to trust and it's possible that
    neither is correct or they're both correct and wrong but in different
    places. So, why not just use the 2nd copy for that reversed linked
    list?

    Alex
    Alexei A. Frounze, Nov 11, 2007
    #17
  18. "Alexei A. Frounze" <> wrote in message
    news:...
    > > This guy has blog on developing a modified FAT for DVR's. Most

    interesting
    > > is summary 2:
    > >
    > > "However, at the same time, the Filesystem team developed an improved
    > > algorithm which should greatly help this situation: The original FAT

    code we
    > > were working with included a single per-open-file 32-bit cached entry of

    the
    > > last cluster accessed within the file, so sequential reads/writes always

    hit
    > > this cache and didn't need to retraverse the FAT chain"
    > >
    > >

    http://blogs.msdn.com/medmedia/archive/2007/01/04/fat-filesystem-performance-issues.aspx
    >
    > Well, what could've been done is either having a copy of FAT with
    > cluster chains in the reverse order or a cache containing intermediate
    > cluster numbers in the chain. To me there's not much benefit of having
    > 2 or more identical copies of FAT. If sh!t happens, tools will have
    > hard times figuring out which one to trust and it's possible that
    > neither is correct or they're both correct and wrong but in different
    > places. So, why not just use the 2nd copy for that reversed linked
    > list?
    >


    Well, I just mentioned it because I thought the cache might've been the
    "FAT24" the OP was talking about... But, I think guy mentioned why they
    didn't do what you suggested just prior to that quote (too many links):

    "Seeking within very large files: When seeking within a file, we have to
    traverse the file's FAT chain to find the desired cluster. As the file grows
    toward the 4GB filesize limit, the number of links that need to be traversed
    gets very large (for 32k clusters, up to 2^17 entries). As mentioned above,
    we partially avoided this issue by not using the full 4GB filesize available
    to us."

    I definately agree that two FAT's can confuse tools _if_ there is no way to
    distinquish which data is valid. I'm sure the point of two FAT's is to
    prevent corruption somehow. But, you'd have to tell me how this is done for
    FAT since I don't know. Filesize? Length of the files' linked list or
    "file chain?"

    The only disk format I've really looked at, decades ago, is the CBM format
    used in the C64's (and reviewed recently...). I've skimmed FAT12. FAT12
    contains the file's linked list in the FAT's, as you know. The CBM format
    distributes the files' linked list across the disk. Each track has a
    pointer to the next track and sector in the chain. The final track has a
    "track value" of zero and "sector value" indicating the offset into the
    sector of the final byte in the file. FAT12 is nice in that the data is
    always 512 bytes, not a few bytes less. Think about potential problems of
    reading 256 bytes from disk and then writing 254 bytes to memory instead of
    256 bytes into memory... The CBM is nice in that the link list is very
    simple, scales easily, is only limited by the size of the BAM (Block
    Availability Map) - a bitmap, and doesn't require increasing sector sizes
    for larger storage capacities, etc. The problem, as I remember it, with the
    CBM formats was when the BAM became corrupt for unknown reasons. You had to
    read in all the sectors and trace all the links to find the start of files
    and recreate the BAM. I'm not sure if the FAT's can be properly recreated,
    if corrupt, since it contains both the track and sector pointers of the
    files' linked list and the sector allocation map. Can it? For the CBM
    format, if the allocation map (BAM) was corrupt, the files' linked list was
    still valid. However, a problem arose when recreating the BAM, both an
    active file and numerous deleted files could point to the same track. If
    you undeleted the "files", you ended up with cross-linked files without any
    way to tell which file chain was good. The solution would've been simple,
    when a file is deleted, clear the track and sector pointers on deleted
    tracks. I'm sure they didn't do that at the time because it would've been
    slow to delete all at once. Today, of course, the FS should wipe the data
    anyway... or at least mark it for wiping later and delete it slowly and
    incrementally.


    Rod Pemberton
    Rod Pemberton, Nov 11, 2007
    #18
  19. Vicne

    Vicne Guest

    On Nov 10, 9:38 pm, "Rod Pemberton" <> wrote:
    > This guy has blog on developing a modified FAT for DVR's. Most interesting
    > is summary 2:
    >
    > "However, at the same time, the Filesystem team developed an improved
    > algorithm which should greatly help this situation: The original FAT code we
    > were working with included a single per-open-file 32-bit cached entry of the
    > last cluster accessed within the file, so sequential reads/writes always hit
    > this cache and didn't need to retraverse the FAT chain"
    >
    > Anyway, MS has produced a number of custom FAT FS, such as ExFAT (WinCE),
    > FATX (Xbox), TFAT. exFAT includes TFAT. TFAT uses multiple copies of the
    > FAT table which I think you mentioned somewhere...


    Hi,

    Thanks for these pointers. Very interesting reading indeed. I'm still
    hoping too this "FAT24" is only there for performance reasons and not
    absolutely needed to read and extract files.

    Kind regards,

    Vicne
    Vicne, Nov 11, 2007
    #19
  20. Vicne

    Vicne Guest

    On Nov 7, 11:36 am, "Alexei A. Frounze" <> wrote:
    > Actually, you could greatly simplify your life by simply taking the
    > raw image of that drive and tweaking the test code in image.c to the
    > drive's geometry. That way you can use the test code to "mount" the
    > image and work with it. The test code is portable C code (no ASM, no
    > BIOS calls to do disk I/O) which you can easily tweak and debug under
    > any OS you want. See imgcpy.c. It was written to copy files to/from a
    > disk image file, almost what you need (you only need to figure out the
    > details of the FS and tweak FAT.c to work with it and duplicated
    > names). Just get fattest and imgcpy compiled and working as-is and
    > then use your image instead of the test one and start digging,
    > tweaking, etc.


    I made progress in that direction yesterday :
    - built imginit and imgcpy,
    - played with them to create a dummy image and simulate the directory
    structure on my PVR (\VIDEO folder with a few files in it)
    - created a new 'imgdir' utility based on 'imgcpy', listing entries in
    a directory in both hex and "interpreted" form
    - replaced the dummy image by an image of the first sectors of the
    actual drive and dumped the entries in the \VIDEO directory

    All of this is working but I'm having difficulties determining what
    data was on the original copy and what was changed by mounting and
    accessing files under various OSes... Moreover, I only exported the
    first 35 Mb of the HD (MBR, BS/BPB, FAT, root dir, VIDEO subdir and
    the few first sectors of actual files) so no actual file extraction
    test can be performed...

    So working with a fresh image is the next step but I'm still stuck
    with a detail: how can I build a bit-by-bit image of a 160 Gb hard-
    drive without mounting it first under Windows...

    Anyway, thanks for providing the FAT toolkit, Alex, it's really great.


    Vicne
    Vicne, Nov 11, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Enrico Migliore

    Re: [Q] FAT Filesystem On MMC Tricks

    Enrico Migliore, Jul 21, 2003, in forum: Embedded
    Replies:
    0
    Views:
    421
    Enrico Migliore
    Jul 21, 2003
  2. Tim Clacy

    Embeddable FAT filesystem

    Tim Clacy, Sep 13, 2004, in forum: Embedded
    Replies:
    24
    Views:
    761
  3. Vicne
    Replies:
    41
    Views:
    1,055
    Vicne
    Nov 2, 2007
  4. Tim Murray
    Replies:
    3
    Views:
    256
    Gregory Weston
    Sep 4, 2004
  5. Replies:
    0
    Views:
    359
Loading...

Share This Page