Large file sharing to, from and within mainland China
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
@coliver said in Large file sharing to, from and within mainland China:
Would something like RDS or VDI work in this instance?
1Mb/s with high latency... I'd guess no.
You're probably right. There doesn't really seem to be a good solution in this event.
-
@scottalanmiller I figured as much. I have a couple synology NAS devices for some scratch space and even then, it is a datastore for VMware. The other is my Veeam backup repo.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@coliver said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
So it looks like there are essentially 2 speeds when dealing with China- The inside China speed and the International speed. Their inside speed is 100Mbps but their average international speed (they refer to it as the International Corridor) is 1Mbps. Yeah... 1!
I have yet to find an optimal way of doing this.
I have asked them to find out how to increase their international bandwidth. I was doing some Googling over the weekend and found that larger companies have this capability but I don't know how they are doing it and how much it costs (probably more than we can spend).
Also, if I can work with a Chinese host for a nextcloud server, setting up some public-facing websites, which nextcloud may or may not be classified as, may or may not require some sort of license from the Chinese government, which could take a month or more or get denied. If I can get that setup, then we are still dealing with an issue where we would need to replicate the data across to a second nextcloud server here. From what I was reading on the nextcloud forums, there is no native way of doing this yet and even if we could, we are still constrained by the 1Mbps average international speed.
The user proposed syncing two Synology NAS devices (1 in China and 1 here), but that introduces all sorts of other headaches on how to get the data shared out to our users here and backed up consistently. As we all know, syncing does not equal a backup.
Would something like RDS or VDI work in this instance?
I have a server, here, running RDS via RDGateway but they are still needing to transfer the large files to, from and within China.
What about a more manual process? Like a dedicated machine inside China that has a NextCloud client? Then they manually manage files inside of China, rather than having them be automatically handled elsewhere? I don't know the end user use case, but if there aren't a lot of users, this is likely pretty simple.
-
@scottalanmiller The problems with that, is that we still need to give and receive the large files from them and also get them backed up.
-
WAN Optimization can help, but with users not in a central location in China, it doesn't help much. There are some serious WAN optimization products out there, but this isn't a great use case for them.
Is there anything that you can do to compress these large files? How big are they, how many and what is the change rate on them? And how much is changed in China vs, in the US?
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller The problems with that, is that we still need to give and receive the large files from them and also get them backed up.
I know and NextCloud will do that.
-
So here is my thought.... You have one machine IN China that syncs to the NextCloud server in the US (or vice versa, but fewer hassles this way.) The Chinese staff grab the file from there and copy it around however they need. They always save changes back to that location, though. So the latest copy is always on the machine syncing to NextCloud. This machine will start sending the updated copy to the US the moment that it is saved, but the Chinese staff get to keep working.
Then the NextCloud server just has to be backed up as usual in the US.
-
@scottalanmiller I had suggested that we can upload the files to a nextcloud server here and send our Chinese users the links so that they can share the data with the factories/vendors but he said that would only work for some of the situations.
EDIT- That would cut out the downloading twice and place the burden on the factories. The reason this isn't ideal is because they sometimes manipulate/edit the files before sending them onto the factories/vendors
The large files are compressed in a zip format.
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller I had suggested that we can upload the files to a nextcloud server here and send our Chinese users the links so that they can share the data with the factories/vendors but he said that would only work for some of the situations.
EDIT- That would cut out the downloading twice and place the burden on the factories. The reason this isn't ideal is because they sometimes manipulate/edit the files before sending them onto the factories/vendors
The large files are compressed in a zip format.
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
Right, my suggestion sounds like it would work fine then since it solves all of those problems. It's your thought that you could share links from Nextcloud that isn't going to work. Take that out and re-read mine again. I think you will find that it will work.
Also, don't use zip it's VERY inefficient. There are much more powerful compression mechanisms that might shave a lot of time off of your transfers.
-
@wrx7m said in Large file sharing to, from and within mainland China:
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
Where do they get modified?
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
So here is my thought.... You have one machine IN China that syncs to the NextCloud server in the US (or vice versa, but fewer hassles this way.) The Chinese staff grab the file from there and copy it around however they need. They always save changes back to that location, though. So the latest copy is always on the machine syncing to NextCloud. This machine will start sending the updated copy to the US the moment that it is saved, but the Chinese staff get to keep working.
Then the NextCloud server just has to be backed up as usual in the US.
What system would I put there? A Windows desktop OS box?
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller I had suggested that we can upload the files to a nextcloud server here and send our Chinese users the links so that they can share the data with the factories/vendors but he said that would only work for some of the situations.
EDIT- That would cut out the downloading twice and place the burden on the factories. The reason this isn't ideal is because they sometimes manipulate/edit the files before sending them onto the factories/vendors
The large files are compressed in a zip format.
We also have video, docs and other files that are used for QC of our products that need to be accessed and stored for our R&D and Execs here .
Right, my suggestion sounds like it would work fine then since it solves all of those problems. It's your thought that you could share links from Nextcloud that isn't going to work. Take that out and re-read mine again. I think you will find that it will work.
Also, don't use zip it's VERY inefficient. There are much more powerful compression mechanisms that might shave a lot of time off of your transfers.
We use 7zip and RAR files a lot, but zip is the most ubiquitous and universal when dealing with the factories and vendors.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller said in Large file sharing to, from and within mainland China:
So here is my thought.... You have one machine IN China that syncs to the NextCloud server in the US (or vice versa, but fewer hassles this way.) The Chinese staff grab the file from there and copy it around however they need. They always save changes back to that location, though. So the latest copy is always on the machine syncing to NextCloud. This machine will start sending the updated copy to the US the moment that it is saved, but the Chinese staff get to keep working.
Then the NextCloud server just has to be backed up as usual in the US.
What system would I put there? A Windows desktop OS box?
Any system. Raspberry Pi, Linux desktop, Windows desktop, Windows server.. whatever you like.
-
@wrx7m said in Large file sharing to, from and within mainland China:
We use 7zip and RAR files a lot, but zip is the most ubiquitous and universal when dealing with the factories and vendors.
You aren't dealing with factories and vendors, you are dealing with internal staff going between US and China. 7Zip cranked to Ultimate would make a lot more sense here.
-
@scottalanmiller said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
We use 7zip and RAR files a lot, but zip is the most ubiquitous and universal when dealing with the factories and vendors.
You aren't dealing with factories and vendors, you are dealing with internal staff going between US and China. 7Zip cranked to Ultimate would make a lot more sense here.
We are dealing with internal staff but they would also need to be able to get these files to the factories via some sort of electronic means. How do my China users get the info to the Chinese factories/vendors (within China). It seems that I am still missing a piece of the puzzle.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller said in Large file sharing to, from and within mainland China:
@wrx7m said in Large file sharing to, from and within mainland China:
We use 7zip and RAR files a lot, but zip is the most ubiquitous and universal when dealing with the factories and vendors.
You aren't dealing with factories and vendors, you are dealing with internal staff going between US and China. 7Zip cranked to Ultimate would make a lot more sense here.
We are dealing with internal staff but they would also need to be able to get these files to the factories via some sort of electronic means. How do my China users get the info to the Chinese factories/vendors (within China). It seems that I am still missing a piece of the puzzle.
They can do anything that they want. Send them in email, provide them from a web server. Deliver then on USB stick. Access them via FTP. Doesn't matter, that's all "after this point." We have to solve getting the data to China first. They can get it around China in numerous ways once it is there.
-
@scottalanmiller Unfortunately, I must also provide them with the means to do that. Maybe they can use one of the Chinese government approved large file transfer services.
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller Unfortunately, I must also provide them with the means to do that. Maybe they can use one of the Chinese government approved large file transfer services.
I understand that you need to do it, I'm just saying that it happens after the NextCloud stage. That's to get the data between the two places. Getting it around in China is likely very easy.
-
@scottalanmiller True but my OP was also for that part
-
@wrx7m said in Large file sharing to, from and within mainland China:
@scottalanmiller True but my OP was also for that part
You can do anything you want once past the NextCloud step. The file is inside China. So the sky is the limit - anything you are allowed and/or willing to do. You have bandwidth at that point or can just walk the files around.
