1. This forum section is a read-only archive which contains old newsgroup posts. If you wish to post a query, please do so in one of our main forum sections (here). This way you will get a faster, better response from the members on Motherboard Point.

Multiple files with same filename on FAT

Discussion in 'Embedded' started by Vicne, Oct 29, 2007.

  1. Vicne

    Vicne Guest

    Hi,

    I'm planning to put the hard disk of my PVR (a Video DVD recorder with
    hard-disk) on an external tray to ease video file moving to my PC.
    Ideally, I would launch a small program to display the HD contents,
    select the sequences I'm interested, and it would then copy them to my
    PC's hard drive for advanced editing and burning. Then I would put the
    drive back into the PVR.

    For testing, I took the drive out, cloned it, and I'm currently
    analyzing its contents.

    It really looks promising : The drive is FAT based (Ghost said FAT 16,
    XP says FAT 32), video files are standard MPG without encryption (they
    play ok in VLC) and there's a "reclist.dat" file listing all recording
    metadata (title, start time, duration, compression mode) that I
    reverse-engineered without too much hassle.

    However, the filesystem obviously is not 100% FAT compliant (for
    example, the last modification timestamp is empty in XP and
    "01/01/1601 01:00" in dos prompt's dir) and the real problem is the
    following :

    When a video file reaches 4 Gb, a second one is created with the
    *same* name (!). For example, "dir" shows :

    01/01/1601 01:00 4.290.772.992 CLIP2.MPG
    01/01/1601 01:00 201.195.520 CLIP2.MPG

    or :

    01/01/1601 01:00 4.290.772.992 CLIP10.MPG
    01/01/1601 01:00 1.055.064.064 CLIP10.MPG

    Due to this inconsistence, I cannot play or copy the file : all I'm
    getting is an error message.

    Does one of you know a simple way (utility, libary, OS) of accessing
    the files directly by their FAT entry ? I'm pretty sure entries are
    contiguous (and even if they're not, finding the correct sequence
    manually would not be a problem).

    As a fallback position, any reliable way to rename the duplicate files
    would be welcome too, although it would probably prevent me from
    putting back the drive in the PVR afterwards unless I undo the
    renaming.

    The program could be in any language and even any OS as I intend to
    place it on a bootable CD that would only be used to make the
    transfer.

    Any hint or help is welcome.

    Vicne


    PS : All recordings (single or multiple files) are accompanied by
    fixed-length .MAP files such as :
    01/01/1601 01:00 4.194.304 CLIP2.MAP
    01/01/1601 01:00 4.194.304 CLIP10.MAP
    Which seem to contain mainly offsets inside the main files. I guess
    the only goal is to allow faster skipping or fast back/forwarding
    inside the video...
     
    Vicne, Oct 29, 2007
    #1
    1. Advertisements

  2. Vicne

    larwe Guest

    1. use dd to make a complete image of the drive
    2. mount it on the loopback device
    3. copy CLIP2.MPG to another location
    4. delete or rename CLIP2.MPG - the FAT code will rename the first
    copy
    5. now concatenate CLIP2.MPG (which will now refer to the second file
    above) with your copied version of the first file
    6. lather, rinse, repeat
     
    larwe, Oct 29, 2007
    #2
    1. Advertisements

  3. Vicne

    R.Wieser Guest

    Vicne <> schreef in berichtnieuws
    ...

    Are you sure that *that* is the cause of the problem ? Can you play/copy
    files that do *not* have a duplicate (in the same directory) ?

    And what did the error-message say. As far as I can tell the first file of
    the two should definitily be accessible, and due to the way an MPG works you
    can view them even if they are not complete.
    That fully depends on the OS you are mounting that drive on.

    Those map-files are there to solve the problem of the file-system not being
    able to handle files larger than 4 GB. They have nothing to do with "fast
    forwarding" or "skipping", but are the equivalent to a FAT (with 4GB
    clusters) : it will probably hold either pointers to the filename-entries,
    or pointers to the first block/sector of the 4GB "clusters".

    Remark : as the PVR uses that method to store largefiles your computer will
    need to use the same method when wanting to store edited MPG's back onto
    that drive again.

    Are you sure that the PVR does not have a working network-conmnection, so
    you can leave the re-assembling (when reading) and dis-assembling (when
    writing) those files to its embedded OS ?

    Regards,
    Rudy Wieser
     
    R.Wieser, Oct 29, 2007
    #3
  4. Vicne

    msg Guest

    Please provide the brand and model of your PVR; there are a great
    many resources on the 'Net for hacking certain units. If you are
    lucky to have certain versions of 'replaytv', you have a great
    many options.

    Regards,

    Michael
     
    msg, Oct 29, 2007
    #4
  5. Vicne

    Vicne Guest

    In fact, concatenating may even be unnecessary as I can join the parts
    in the video editing program. I'd be happy if I could just *read* them
    separately.
    So that's definitely an interesting path. Indeed, working on an
    *image* instead of a *clone* would be much simpler and I could try to
    analyse the drive without the OS trying to "repair" it. (I tried
    mounting it on a Red Hat box but it hung, so I tried with XP which was
    successful).
    The only drawback is that imaging the 160Gb drive takes around 1h30,
    but well...

    Thanks a lot for the suggestion !

    Vicne
     
    Vicne, Oct 29, 2007
    #5
  6. Vicne

    Vicne Guest

    Yes, definitely.

    As I said, I could play other video files in VLC without any problem.
    I didn't watch a clip from start to end, but I can open them, see the
    video, hear the audio (in sync), and I can skip forward and backward
    OK. I tried with three or four without any problem, and the only ones
    that cannot be read are the "multiple" ones...
    I admit I was surprised that it was so simple (almost compliant with
    FAT, no proprietary video format, no encryption...)
    You're right, I should have noted the exact message (I'll do it next
    time), but it's a standard uninformative XP message like 'Error
    reading file' with only an OK button. The explorer doesn't crash, it
    simply reports the error. I also tried accessing the file under
    Command Prompt, but I'm also getting an error.

    I was surprised too that none of the two files was playable. I
    expected that one would be and the other not, or that double-clicking
    any of them would play the same video, but no : only an error message.
    That was XP, but I'll use whatever OS allows me to copy the files
    over, booting it from a USB stick for example.
    That's very interesting ! Do you have a link to more information about
    the structure of these 'MAP' files ? I quickly googled 'map files 4
    Gb' but nothing seems relevant at first. Might look further into that.
    Fortunately, I don't want to put anything back to the hard disk. I'll
    edit and burn DVDs and it will be all.

    To give you more info, I'm in the process of converting a whole bunch
    of VHS tapes to DVD, and VHS to PVR via S-VIDEO gives the best quality
    so far. Unfortunately, this PVR has no editing feature, and moreover
    its DVD burner is complete crap : it only accepts certain DVD brands,
    it only allows about 15 DVD-RW write/wipe cycles after which it
    considers it's corrupt, and it only works when PVR is cold (burning 2
    DVDs in a row fails inevitably).
    Yes, I'm sure unfortunately (or maybe the chipset does but it has no
    external network connection). It's all Cirrus Logic based (CS98301
    which I guess is similar to the 98300 they talk about on
    http://www.cirrus.com/en/press/releases/P399.html). The only external
    digital connection is Firewire, but it's only there to control a
    camcorder and import video to the internal hard drive.

    Many thanks for the insightful help. It's much appreciated.

    Vicne
     
    Vicne, Oct 29, 2007
    #6
  7. Vicne

    Vicne Guest

    Well, it's a cheap model branded "Oasis-Media", model DVD-9407. I
    bought it in Belgium but it seems it was mainly sold in the
    Netherlands. A firmware update can even be found on
    http://www.oasis-media.be/rma/dvd/DVD-9407.html
    I'm afraid it's not. All I can say is that it's Cirrus Logic based,
    but even the exact chip reference (CS98301) is not easy to find
    online...

    Thanks for the help anyway.


    Vicne
     
    Vicne, Oct 29, 2007
    #7
  8. Vicne

    Vicne Guest

    After further investigation, it seems it was made by Sampo -
    http://www.sampo.com.tw/english/index.asp

    Unfortunately, all references I now find on the net point to the
    defunct area450.com website, part of which can still be read via the
    wayback machine on http://web.archive.org.

    My version has a 160 Gb hdd but looks exactly the same as the DV-RH680
    - see http://web.archive.org/web/20070202172538/www.area450.com/recorders/rh680.html
    - but there isn't much information about it online

    Reports say it's equivalent to the Sampo DVE-R8HP :
    http://web.archive.org/web/20070102160237/http://www.area450.com/recorders/r8hp.html
    According to that page, it was also sold by the names Medion MD42183
    (Europe), Micromaxx MM80103 (Germany), Procaster HDRW80GB (Finland),
    Tevion MD80120 (Australia)

    But well, I don't think it's getting me very far...

    Vicne
     
    Vicne, Oct 29, 2007
    #8
  9. Vicne

    R.Wieser Guest

    That was what I was not all-that sure about : that the drive-geometry and
    file-system was recognised by that other computer.
    You could try to rename the double file. If you're lucky you won't get an
    error-message, and only one of the two gets renamed. After that you should
    be able to copy them to your XP-computers harddrive, where you could than
    use an utility to stich the files back together again. If you're *very*
    lucky a simple "copy file1.mpg /b file2.mpg /b resultfile.mpg" would work.

    [snip]
    I'm sorry, no. Its just some info I picked up somewhere when I was
    searching for answers related to a problem of my own, allso related to an
    embeded OS and a hard-disk (a LAN-drive actually).

    [snip]
    You're welcome.
     
    R.Wieser, Oct 29, 2007
    #9
  10. Vicne

    Matt Guest

    What's the format of the map file? Have you tried running DOS, and
    simply reading the sector that equates to the first entry in the FAT at
    the offset given in the map file? If this is the start of the file, it
    should have a valid MPG4 header at the start. A quick way of finding out
    what's where would be to run a defrag program on the disc. This will
    show you which parts of the disc are in use. (Obviously stop before you
    actually start moving stuff) and you can then figure how these sections
    of the disc relate to the numbers in the map file.

    Matt
     
    Matt, Oct 29, 2007
    #10
  11. It should be safer to first rename all instances of CLIP2.MPG. With all
    this caching etc in today's operating systems, I wouldn't blindly assume
    that the "copy" command accesses the same instance than the following
    "delete" (at least, my ISO9660 implementation wouldn't guarantee that in
    case of an error like this, and I would use the same caching trick again
    when implementing FAT).


    Stefan
     
    Stefan Reuther, Oct 29, 2007
    #11
  12. Vicne

    Vicne Guest

    Well, don't know exactly. All map files are exactly 4 kB long, no
    matter the clip size.
    I zipped one and uploaded it here : http://www.megaupload.com/?d=HWR38923
    (167 kB).
    It's mostly empty as you can see.
    Note that my description of it as offsets inside the video file was
    only a guess...
    No, but that's a good idea.
    Can you advise a DOS utility to perform such direct disk reading ?
    Mmmh, that's a good idea. I'll try that indeed.

    Thanks for the tip.


    Vincent.
     
    Vicne, Oct 29, 2007
    #12
  13. Vicne

    Vicne Guest

    Yes, I also prefer renaming instead of deleting.
    I'll try that to see if renaming is allowed or if it also gives an
    error.

    Thanks for the help.


    Vicne
     
    Vicne, Oct 29, 2007
    #13
  14. Vicne

    Vicne Guest

    Yes, I should have tried earlier, but I did all I could not to modify
    anything on the clone, to allow further investigation...
    And as you say, I could get an error-message. I'll have to try.
    Yeah, that's not even needed as any editing software will be able to
    join the pieces.
    OK, I see.

    Thanks for the help.

    Vicne
     
    Vicne, Oct 29, 2007
    #14
  15. Vicne

    larwe Guest

    You REALLY need to think that through, Stefan. If cache coherency is
    so screwed up, then there is no way the operating system could
    possibly work. Reads from a drive where write-behind cache is active
    will read from the cache, not the media.

    It doesn't really matter anyway - the important point is, all the FAT
    code I've ever read, written or tested will only search directories
    until it hits the first instance of the filespec. So "mv CLIP2.MPG
    CLIP2.MP0" will only change the first one.
     
    larwe, Oct 29, 2007
    #15
  16. Vicne

    Jim Stewart Guest

    This is where you lost me.

    FAT16 uses a directory table and one or more
    FATs. The entries in the directory table point
    to offsets into the FATs. To the best of my
    knowledge, there is no filename info in the FATs.
     
    Jim Stewart, Oct 29, 2007
    #16
  17. Vicne

    Vicne Guest

    That makes sense indeed, and I think in any case renaming one of the
    files is the safest way to proceed.

    I was wondering also : I think a solution would be to mimic what a
    file recovery utility would do.
    Their job is to decode a source filesystem, detect deleted entries and
    recover the files they pointed to to another drive. What I want to do
    is very similar, except I'm interested in duplicate entries instead of
    deleted ones.

    I found PhotoRec - http://www.cgsecurity.org/wiki/PhotoRec - but
    unfortunately, PhotoRec is precisely different in that it does care
    about filesystem structure and only scans sectors for known file
    "signatures".

    Do you know of a "file recovery" (or maybe "undelete") utility for DOS
    with available source code ?

    Thanks very much.

    Vicne
     
    Vicne, Oct 29, 2007
    #17
  18. He didn't ask to read the filename, but the file. All you need is
    the first FAT entry, and a knowledge of the FAT structure, and you
    can follow the FAT chain to get all block numbers for the file.
    The blocks aren't guaranteed to be contiguous. The only missing
    information is how much of the last block is used.

    But really, the best solution here is to open the raw drive in a
    disk editor, search for the filename, and change each name to
    something unique. Not hard, and though I haven't done it for
    years, such programs are out there.

    Clifford Heath.
     
    Clifford Heath, Oct 29, 2007
    #18
  19. Vicne

    larwe Guest

    Deleted entries in a FAT filesystem are marked as deleted by changing
    the first character of the filename to a special magic byte. All DOS
    undelete utilities look for this signature. Other utilities (which
    look for deleted files in unused space after the directories are
    destroyed) work by looking for specific data signatures such as JPEG
    headers.

    If you want sourcecode for a FAT16/32 reader that can do what you want
    (with some small glue you will have to provide), I wrote one:
    http://www.zws.com/products/dosfs/
     
    larwe, Oct 30, 2007
    #19
  20. Vicne

    Vicne Guest

    You're right.

    I'm afraid my sentence was wrong. I didn't mean "FAT entry" but
    "Directory entry".
    I mean : If I could somehow navigate the disk with a file manager that
    showed raw directory entries instead of filenames, I could probably
    disambiguate which file I want to work with.

    Sorry for the confusion.

    Vicne
     
    Vicne, Oct 30, 2007
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.