As I hinted in a previous thread I'm trying to copy full RISC OS hard disc images to my Linux machine. Thanks to everyone I've got the reading the image sorted (in my head) but am trying to think the best way to transfer them.
What I want to do is be able to transmit the image from RISC OS reading the disc and it end up as a file at the Linux end. Since this file is going to be bigger than 4GB I can't use the usual RISC OS file transfer methods. It also needs to be as fast as possible, and ideally as much as possible done by hacking existing software (probably something in C I can compile for RISC OS).
I had a go with a TFTP client/server I hacked to accept a 64 bit block count (instead of 16 bit). I'd hoped its use of UDP would make it fast, but it's too slow because it too simplistic: it waits for a round-trip for each block, and UDP limitations mean that's limited to a few KB.
Any suggestions? Really I want a file transfer protocol that supports >4GB; uses UDP if it does it efficiently and has support for dropped packets, otherwise TCP; and has a client I can compile for RISC OS and easily replace the file reading code with sector reading. Or is there a pre-existing RISC OS file transfer application with source that will cope with such large files even if it never sees them normally?
I could write a filing system that presented my disc as many 1GB files but I'd rather not have "FTP error: retrying file" if there's a transmission problem. And writing a filing system sounds a bit too much like hard work.
Theo Markettos <theom+n...@chiark.greenend.org.uk> wrote: > What I want to do is be able to transmit the image from RISC OS reading the > disc and it end up as a file at the Linux end. Since this file is going to > be bigger than 4GB I can't use the usual RISC OS file transfer methods. It > also needs to be as fast as possible, and ideally as much as possible done > by hacking existing software (probably something in C I can compile for RISC > OS).
You could hack the Sunfish code. Use NFS3 which supports 64bit filesizes, and for each open filehandle maintain a current offset. Add the current offset to the offset requested normally. In the program generating the image, write the first 2GB, then reset the file pointer to 0 and write the second 2GB, and so on. In Sunfish, detect a write to 2GB followed by a write to 0, and then increment the current offset by 2GB.
> As I hinted in a previous thread I'm trying to copy full RISC OS hard disc > images to my Linux machine. Thanks to everyone I've got the reading the > image sorted (in my head) but am trying to think the best way to transfer > them.
> What I want to do is be able to transmit the image from RISC OS reading the > disc and it end up as a file at the Linux end. Since this file is going to > be bigger than 4GB I can't use the usual RISC OS file transfer methods. It > also needs to be as fast as possible, and ideally as much as possible done > by hacking existing software (probably something in C I can compile for RISC > OS).
> I had a go with a TFTP client/server I hacked to accept a 64 bit block count > (instead of 16 bit). I'd hoped its use of UDP would make it fast, but it's > too slow because it too simplistic: it waits for a round-trip for each > block, and UDP limitations mean that's limited to a few KB.
I've done quite a bit of work relating to this on Linux, but not RISC OS. But you probably want to use TCP, with as large buffers as possible. Depending upon your protocol, you may want to use TCP_NODELAY as well, to disable the Nagle algorithm. If you were sending from Linux, I would also say to use sendfile(), although that doesn't make much sense in RISC OS, since this greatly reduces CPU usage.
-- Peter Naulls - pe...@chocky.org | http://www.chocky.org/ --------------------------------------------------------------------------- - RISC OS Community Wiki - add your own content | http://www.riscos.info/
> As I hinted in a previous thread I'm trying to copy full RISC OS hard disc > images to my Linux machine. Thanks to everyone I've got the reading the > image sorted (in my head) but am trying to think the best way to transfer > them.
> What I want to do is be able to transmit the image from RISC OS reading the > disc and it end up as a file at the Linux end. Since this file is going to > be bigger than 4GB I can't use the usual RISC OS file transfer methods. It > also needs to be as fast as possible, and ideally as much as possible done > by hacking existing software (probably something in C I can compile for > RISC OS).
> I had a go with a TFTP client/server I hacked to accept a 64 bit block > count (instead of 16 bit). I'd hoped its use of UDP would make it fast, > but it's too slow because it too simplistic: it waits for a round-trip for > each block, and UDP limitations mean that's limited to a few KB.
> Any suggestions? Really I want a file transfer protocol that supports > 4GB; uses UDP if it does it efficiently and has support for dropped > packets, > otherwise TCP; and has a client I can compile for RISC OS and easily > replace the file reading code with sector reading. Or is there a > pre-existing RISC OS file transfer application with source that will cope > with such large files even if it never sees them normally?
> I could write a filing system that presented my disc as many 1GB files but > I'd rather not have "FTP error: retrying file" if there's a transmission > problem. And writing a filing system sounds a bit too much like hard work.
Perhaps you can simply read block by block by a BASIC program and put the output over PipeFS or directly to "nc" (netcat, available from the UNIX porting project AFAIR).
On UNIX side you must recieve data using also "nc". I assume that you simply have to spool the received data into a file.
I didn't know whether it works and whether there are some problems with some control codes which must be wrapped/extracted but perhaps it would be worth to try it. You can check by transferring single binary files, transfer them in classical way parallel and compare them afterwards.
On 28 May 2006 Peter Naulls <pe...@chocky.org> wrote:
> I've done quite a bit of work relating to this on Linux, but not RISC > OS. But you probably want to use TCP, with as large buffers as > possible.
Agreed, TCP rather than UDP is derfinatley best for large transfers.
> Depending upon your protocol, you may want to use TCP_NODELAY > as well, to disable the Nagle algorithm.
Only if transmitting small packets, such as one or two bytes commands, otherwise it slows down transfers, particularly where you send a length word followed by the bulk of the data in seperate send call.
> If you were sending from Linux, I would also say to use sendfile(), > although that doesn't make much sense in RISC OS, since this greatly > reduces CPU usage.
I have some very trivial C programs that can be used to send a block of data down a TCP stream and save it out as a file at the other end. Although I suspect it would need enahancing to enable continuation after interruption with the size of data involved - you dont want to have to statr from the beginning again with multiple GB.
Alex Waugh <a...@alexwaugh.com> wrote: > You could hack the Sunfish code. Use NFS3 which supports 64bit > filesizes, and for each open filehandle maintain a current offset. Add > the current offset to the offset requested normally. In the program > generating the image, write the first 2GB, then reset the file pointer > to 0 and write the second 2GB, and so on. In Sunfish, detect a write to > 2GB followed by a write to 0, and then increment the current offset by > 2GB.
That's a good idea... I'll have a look at it. Out of interest (not having looked at the code and seen if it's possible) would it make more sense to have a SWI which sets the 2GB pointer for a particular open file? That might it more generally useful for other people who wanted to write >2GB files with programs. Or some other more general way to indicate which 2GB block is being written to (your method is general but there's no way to have random access to 2GB blocks, not that I need it).
Peter Naulls <pe...@chocky.org> wrote: > I've done quite a bit of work relating to this on Linux, but not RISC > OS. But you probably want to use TCP, with as large buffers as > possible. Depending upon your protocol, you may want to use TCP_NODELAY > as well, to disable the Nagle algorithm. If you were sending from > Linux, I would also say to use sendfile(), although that doesn't make > much sense in RISC OS, since this greatly reduces CPU usage.
Thanks, I'll bear those in mind. I'd particularly never heard of sendfile() which sounds quite useful, even if it doesn't work on RISC OS (I could conceive an evil implementation that reads directly from filing system buffers into mbufs ready for transmission, but I'm not going to do it).
Thomas Milius <Thomas-Mil...@t-online.de> wrote: > Perhaps you can simply read block by block by a BASIC program and put the > output over PipeFS or directly to "nc" (netcat, available from the > UNIX porting project AFAIR).
> On UNIX side you must recieve data using also "nc". I assume that you > simply have to spool the received data into a file.
> May be that nc is slow.
I think that would probably work, but the issue would be that PipeFS is too slow. That's two additional copies of the data that need to be made. It would be more efficient to modify the nc code to read disc blocks rather than from stdin. In which case it's possibly simpler to write something which opens a TCP socket and throws data at it, and something that spools everything it receives to a file. But I don't seem to be able to get much above 100KB/s with TCP (FTP downloads etc) on my Risc PC, which is slightly disappointing when in Linux it quite happily manages 600KB/s. I was wondering if there was a way to speed this up (Peter's tips might help).
Theo Markettos <theom+n...@chiark.greenend.org.uk> wrote: > Alex Waugh <a...@alexwaugh.com> wrote: > > You could hack the Sunfish code. Use NFS3 which supports 64bit > > filesizes, and for each open filehandle maintain a current offset. Add > > the current offset to the offset requested normally. In the program > > generating the image, write the first 2GB, then reset the file pointer > > to 0 and write the second 2GB, and so on. In Sunfish, detect a write to > > 2GB followed by a write to 0, and then increment the current offset by > > 2GB.
> That's a good idea... I'll have a look at it. Out of interest (not having > looked at the code and seen if it's possible) would it make more sense to > have a SWI which sets the 2GB pointer for a particular open file? That > might it more generally useful for other people who wanted to write >2GB > files with programs. Or some other more general way to indicate which 2GB > block is being written to (your method is general but there's no way to > have random access to 2GB blocks, not that I need it).
You'd have to convert from a fileswitch handle using OS_FSControl 21, but otherwise that would be fairly easy.
In message <df050b2e4e.dr...@druck.freeuk.net> druck <n...@druck.freeuk.com> wrote:
> On 28 May 2006 Peter Naulls <pe...@chocky.org> wrote: > > I've done quite a bit of work relating to this on Linux, but not RISC > > OS. But you probably want to use TCP, with as large buffers as > > possible.
> Agreed, TCP rather than UDP is derfinatley best for large transfers.
Yes, that's what I said.
> > Depending upon your protocol, you may want to use TCP_NODELAY > > as well, to disable the Nagle algorithm.
> Only if transmitting small packets, such as one or two bytes commands, > otherwise it slows down transfers, particularly where you send a length word > followed by the bulk of the data in seperate send call.
As I said, it depends upon the algorithim. In some cases, use of TCP_CORK is also appropriate, but that is probably not supported on RISC OS.
Is there an echo in here?
-- Peter Naulls - pe...@chocky.org | http://www.chocky.org/ --------------------------------------------------------------------------- - RISC OS Community Wiki - add your own content | http://www.riscos.info/
Theo Markettos wrote: > Thomas Milius <Thomas-Mil...@t-online.de> wrote: >> Perhaps you can simply read block by block by a BASIC program and put the >> output over PipeFS or directly to "nc" (netcat, available from the >> UNIX porting project AFAIR).
>> On UNIX side you must recieve data using also "nc". I assume that you >> simply have to spool the received data into a file.
>> May be that nc is slow.
> I think that would probably work, but the issue would be that PipeFS is too > slow. That's two additional copies of the data that need to be made. It > would be more efficient to modify the nc code to read disc blocks rather > than from stdin. In which case it's possibly simpler to write something > which opens a TCP socket and throws data at it, and something that spools > everything it receives to a file. But I don't seem to be able to get much > above 100KB/s with TCP (FTP downloads etc) on my Risc PC, which is slightly > disappointing when in Linux it quite happily manages 600KB/s. I was > wondering if there was a way to speed this up (Peter's tips might help).
In the past I've managed to transfer data between machines via TCP/IP in Python. Worth a look.
> Thomas Milius <Thomas-Mil...@t-online.de> wrote: >> Perhaps you can simply read block by block by a BASIC program and put the >> output over PipeFS or directly to "nc" (netcat, available from the >> UNIX porting project AFAIR).
>> On UNIX side you must recieve data using also "nc". I assume that you >> simply have to spool the received data into a file.
>> May be that nc is slow.
> I think that would probably work, but the issue would be that PipeFS is too > slow. That's two additional copies of the data that need to be made. It > would be more efficient to modify the nc code to read disc blocks rather > than from stdin. In which case it's possibly simpler to write something > which opens a TCP socket and throws data at it, and something that spools > everything it receives to a file. But I don't seem to be able to get much > above 100KB/s with TCP (FTP downloads etc) on my Risc PC, which is slightly > disappointing when in Linux it quite happily manages 600KB/s. I was > wondering if there was a way to speed this up (Peter's tips might help).
Was this using TCP or UDP - I seem to recall that the problem with ShareFS was caused by its use of UDP, exposing a bug in the stack. You might be hitting the same bug.
In article <WSj*w4...@news.chiark.greenend.org.uk>, Theo Markettos <theom+n...@chiark.greenend.org.uk> wrote:
> Peter Naulls <pe...@chocky.org> wrote: > > I've done quite a bit of work relating to this on Linux, but not RISC > > OS. But you probably want to use TCP, with as large buffers as > > possible. Depending upon your protocol, you may want to use TCP_NODELAY > > as well, to disable the Nagle algorithm. If you were sending from > > Linux, I would also say to use sendfile(), although that doesn't make > > much sense in RISC OS, since this greatly reduces CPU usage.
> Thanks, I'll bear those in mind. I'd particularly never heard of > sendfile() which sounds quite useful, even if it doesn't work on RISC OS > (I could conceive an evil implementation that reads directly from filing > system buffers into mbufs ready for transmission, but I'm not going to do > it).
No idea if this is a viable idea, but you might be able to bodge your data over to a Linux box by fiddling with the sources to Moonfish and simply serving the file from a share on the RISC OS box? I'd imagine that Moonfish isn't limited to 32bit file sizes in itself, rather by the underlying limitations of FileCore.
"Ste (news)" <steve.DEL...@revi11.plus.com> wrote: > No idea if this is a viable idea, but you might be able to bodge your data > over to a Linux box by fiddling with the sources to Moonfish and simply > serving the file from a share on the RISC OS box? I'd imagine that Moonfish > isn't limited to 32bit file sizes in itself, rather by the underlying > limitations of FileCore.
In fact that's almost what I've done, except with Sunfish. I've added a SWI which, when called with a file handle, will move the pointer at the server end onto the second, third, etc 2GB chunk so it can store up to 16 exabytes. Unfortunately I've got some hassles about getting an NFS3 server running on my Linux machine[1] so haven't been able to test it yet. I've been trying various servers and mostly tying the portmapper in knots. I'm talking to the Linux NFS project mailing list to sort that end out before I can test my changes :)
[1] I need my existing NFS 2 user-mode server since it's the only one that will re-export mounts, meaning I can mount something on my Linux machine then export the parent of that mount to RISC OS, which means I can use Linux as a proxy to mount things using filing systems that aren't supported on RISC OS (like sshfs).
|
