My DreamHost Subversion now can't commit/add any new file over ~.5GiB or .5GB

apps

#1

problem LQATUZ post LQDD1Q
I’m having the problem described in the subject (“My DreamHost Subversion now can’t commit/add any new file over ~.5GiB or .5GB”). Anyone else know-of or experiencing this? As it’s certainly got me stuck; and it looks like it could be affecting others, potentially everyone on DreamHost.

And based on (the cryptic error message which occurs and my frustrating experiences) which I present below, possible causes I can see include, from most to least likely, are: [list=1]
[]Possibly DreamHost quietly added (else notably reduced) a limit on Subversion checkin size (for the whole checkin else for a file), seemingly .5GiB (a typical value) else .5GB (a somewhat typical value) when before I was having no trouble at over 1GB, and a limit which manifests itself as a cryptic error (described below) which also oddly doesn’t appear until after 24 to 87MiB are transferred. If DreamHost did this, based on my experiences, they must have done this between 2011.06.04 (when I added-via-commit a 1.03GiB file with no problem) to 2011.08.21 (when now I can’t commit any file .51GiB or larger). And if DreamHost did this, they would have done this either directly (by somehow setting some explicit limit) else indirectly (by say changing one of the involved server programs, which somehow then imposed (as added) this limit as say a new “feature”).
[
] DreamHost’s software is has become out-of-date else inconsistent. While indeed DreamHost software is mostly out-of-date (as most notably, my account (somewhat disappointingly) has “svn, version 1.5.1 (r32289)” even though (the current version 1.6 has been out for 2.4years says http://subversion.apache.org/docs/release-notes ), and the OS (Debian) behind at least 6mo, so I wouldn’t be surprised then if the web server (Apache) was as well), DreamHost’s software being out-of-date seems unlikely to be the problem as DreamHost’s supplied Subversion and some/all this related software always been out-of-date apparently due to the problem detailed here on the Dreamhost Debian wiki page yet still I was able to easily add-via-commit a 1.03GiB file 2mo ago as mentioned; so while still I would very much like DreamHost to update the software of me (and of everyone else who might need it, as being 2.4years behind is kind of scary, especially in IT, and especially when it comes to storing precious sources, what Subversion does), I still don’t strongly suspect DreamHost’s out-of-date software being the present cause of this problem, unless say it is from some (older?) Subversion bug (if there is one) where the (supposedly unlimited) file/checkin size gets limited as the repository grows, or from some fringe-combo-version-incompatibility-bug (if there is one) from say DreamHost updating some other software on my server (if they did) while still letting others (as Subversion) lag 2.4years behind.
[] Possibly something corrupted in my SVN repository (note I have only tried files over .5GiB in this one repository, since these files are large and Subversion keeps files forever, so this is my Subversion repository for oversized items & collections), but this repository has always worked before, including always working fine for at least for all files under ~.5GiB or GB, and before also working fine for the 1.03GiB file I mentioned 2mo ago , and I’m experiencing this failure identically on 2 of 2 Subversion clients, so I kind of doubt it is my repository.
[
] It’s not seemingly my client computer & it’s SVN checkout, as the problem occurs identically on 2 of 2 different ones as mentioned.
[] It’s not seemingly my Internet connection as the problem occurs identically on 2 of 2 totally separate Internet connections.[/list]
So in response to my #1 suspect, Did DreamHost (or the Apache web server or involved server software) suddenly add-else-notably-lower a file-size or checkin-size limit? While I certainly hope not, it unfortunately looks like it, and if so, then
not only was that done without a meaningful error message when the limit is reached (so costing me 2 days of time making & analyzing these tests),
it also appears to have been done without any public warning & announcement & documentation, as:[list=1]
[
] Google Search(DreamHost +Subversion OR +SVN size OR GiB OR MiB limit) currently finds for me:[list=a] [] Nothing specifically mentioning any limits. Notably it does NOT find any size limits mentioned where one would expect them to be mentioned: on the official page http://wiki.dreamhost.com/Subversion .
[
] http://discussion.dreamhost.com/thread-67887-post-67909.html#pid67909 saying in 2007 that the servers (including those hosting Subversion) had a 2GiB file Apache web server limit, which 4x higher than what I’m experiencing here.[/list]
[] Google Search(+Subversion OR +SVN size OR GiB OR MiB limit) currently finds for me:[list=a] [] 1st the official document saying “A nice feature of Subversion is that by design, there is no limit to the size of files it can handle. …”.
[] 2nd this useful thread http://www.svnforum.org/threads/39795-Is-there-any-inherent-Subversion-repository-size-limit also saying Subversion itself imposes no practical limits but can be limited to generally 2GiB but only if the filesystem isn’t properly upgraded, and while that is also not good, it is still 4x larger than the ~.5GiB limit I am experiencing.[/list][/list]
So what’s the problem I’m experiencing exactly? Subversion commits now get their connection mysteriously aborted when the commit contains (adds) any new file over ~.5GiB, including:[list=1]
[
] This didn’t happen 2mo ago (when I committed a new 1.03GiB file without a hitch), though (as noted below) I did get not-the-same-but similar failure 5mo ago when trying to add-via-commit a 2.32GiB file.
[] This failure doesn’t occur at the start nor end of commit, but oddly after transmitting the first ~25 to 87MiB
[
] They error message is upsettingly not clear being vague, phrased either:[list=a]
[] From TortoiseSVN 1.6.16 (latest) “
Command: Commit
Adding: …\L9LIBO_LIP2I8_oversized_L9QDKC\Asus_Eee_PC_Vmx__LQ7F9N\LQ7KPS\LQ7KPS-s002.vmdk application/octet-stream
Sending content: …\L9LIBO_LIP2I8_oversized_L9QDKC\Asus_Eee_PC_Vmx__LQ7F9N\LQ7KPS\LQ7KPS-s002.vmdk
Error: Commit failed (details follow):
Error: PUT of
Error: ‘/L9LIBO/!svn/wrk/8e3e9e75-a7fe-6042-aa64-761d7b26c45a/_oversized_L9QDKC/Asus_Eee_PC_Vmx__LQ7F9N/LQ7KPS/LQ7KPS-s002.vmdk’:
Error: Could not send request body: An existing connection was forcibly closed by the
Error: remote host.
Error: (http://subdomain1_alias.dreamhosters.com)
Finished!:
” after transmitting just 87.38MiB of this 2,143,617,024 byte file. OR
[
] from Cygwin svn 1.6.17 (latest) “
Transmitting file data .svn: Commit failed (details follow):
svn: PUT of ‘/L9LIBO/!svn/wrk/258318e6-cc9a-11e0-aa24-2bb4316230e0/_oversized_L9QDKC/Asus_Eee_PC_Vmx__LQ7F9N/LQ7KPS/LQ7KPS-s011.vmdk’: Could not send request body: Software caused connection abort (http://subdomain1_alias.dreamhosters.com)
” after transmitting ~2min worth (~25MiB) of this 547,815,424 byte file.[/list][/list]
As suggested, this failure doesn’t occur everywhere for me, but so far only with files over ~.5GiB, and specifically:[list=1]
[] This problem has NOT occurred (instead did a checkin with no problem):[list=a]
[
] On all other checkins within at least that last 2 months, which have been as big as ~126.63MiB (a recent diff on this Outlook folder LM3NRK/).
[] On 2011.06.04 (just 2mo ago), a new file of 1.03GiB to this same repository (file LM3NRK/outlook_LCC5FC.ost, so also to another folder).
[
] Within this particular source folder where I’m now seeing problems, folder LQ7KPS/ : all files from 0 bytes to 461,832,192 bytes(=.43GiB=.46GB) checkin no-problem.[/list]
[] Indeed, so far all files having this failure-to-checkin problem have been:[list=a]
[
] On committing every file 547,815,424 bytes(=.51GiB=.54GB) or larger, where:
[] On commits of new files (so on adding files via commit) but not on file diffs (probably since I haven’t had any big diffs over ~.5GiB), though even if the problem is just for adds, it is still a serious problem (as hopefully one wouldn’t have to do a workaround of explicit file splitting & reassembly).
[
] Within one particular source folder, folder LQ7KPS/ – but I don’t think this folder is the problem for, as mentioned, all is files from 0 to .43GiB HAVE added-via-commit without a problem.[/list]
[] As mentioned, the failure occurs not at the start nor end of the transfer, but right after the first 24.13MiB to 87.38MiB.
[
] When a commit is reattempted, the error occurs not at exactly the same point in the transfer but very close, within +/- .3MiB. This repeatability suggests to me that this problem is not a flaky hardware (or software), but a situation where the (server=recipient) reaches some (likely software) limit (seemingly .5GiB) so reports back to the client to abort, which then immediately aborts within +/- a few packets transmitted.
[] The client computer probably doesn’t matter (as happened identically on both a Win7 (Starter & Home Premium)).
[
] Internet connection doesn’t seem to matter (as happened on office Wi-Fi, office Ethernet-to-router, and Starbucks Wi-Fi)
[] Subversion client doesn’t seem to matter (as happened via latest TortoiseSVN and Cygwin svn, using each to do both the add-op and the checkin-op)
[
] About 5mo ago, I got not the same but similar TortoiseSVN error (only different in the last 1/2 of error message, and only somewhat: “Error: Could not send request: An established connection was aborted by the software \ Error: in your host machine.”) trying to add-via-commit a 2.32GiB file (file X15-65804.iso commit LI983N on 2011.03.18).[/list]
Google Search(SVN Commit failed SVN wrk send request body connection forcibly closed) led me to number of possible fixes for this, such as http://code.google.com/p/support/issues/detail?id=2726 suggesting doing a Subversion Cleanup first, updating the router’s firmware, and trying with a command-line tool, but I’ve done all 3 things (including not only updating the router’s firmware but connecting to the router via Ethernet instead of 802.11g) and none of them seem to improve nor change anything.

To my notable frustration, Google Searching for the error message as I explain above, plus even for its likely cause as I explain above, I have not been able to find any explanations, why then I make this post.

Today or tomorrow I will be asking DreamHost support to reply to this.[list=1]
[]And if DreamHost indeed does have any Subversion file size limit (such as mere .5GiB), for me at least, that would be[list=a]
[
] a serious drawback in choosing & using & recommending DreamHost (especially when DreamHost’s inexpensive Subversion hosting was the #1 reason I picked them), and
[] seemingly false-advertising as (DreamHost clearly advertises “unlimited” storage including definitely up to 50 GB or GiB, notably here http://dreamhost.com/unlimited-policy ).[/list]
[
]The only size limit which would seem acceptable would be one truly imposed by the software if this-couldn’t be fixed/enlarged by an upgrade, or a limit at the highest practical limit, notably at least enough to distribute (as a single file) the largest common removable media, which currently is (46.6GiB per http://en.wikipedia.org/wiki/Blu-ray_Disc#Physical_media ).
[]So if DreamHost indeed has such limits, then hopefully they will now:[list=a] [] clearly & publically announce any limits, and any limits they did have, including where possible in the error messages when the limit is reached
[*] most especially, remove any practical limits as I describe here.[/list][/list]
But, especially as this would seem to affect anyone and the error message is not clear (and maybe someone will know about it), I also post this here report on the public DreamHost forums, so your thoughts & suggestions appreciated, too.

[hr]Aside: Here (& officially), GiB is 2^30=1024^3 bytes, as defined by http://en.wikipedia.org/wiki/Gibibyte ; so then “GB” is really 2^9=1000^3 bytes, as defined by http://en.wikipedia.org/wiki/Gigabyte . Not knowing of the “Gi” symbol (and in general the “i” insert) until now, until now I (as many people, including MS Windows 7) have miscalled “GiB” (and “MiB, KiB”, etc) as “GB” (and “MB”, “KB” respectively); but since 1024^3 (“Gi”) is not quite the same (slightly larger) than 1000^3 (classic “G”=“Giga”), it makes sense that it has its own symbol “Gi”, and indeed http://en.wikipedia.org/wiki/Blu-ray_Disc#Physical_media alerts me of this, so I will be more careful from now on and not use “G” when I mean “Gi”. Indeed per http://en.wikipedia.org/wiki/Binary_prefix , I see one should insert the “i” (as “Ki, Mi, Gi, Ti” instead of “K, M, G, T” respectively) whenever one really means the binary multiplier 1024^n instead of the decimal multiplier 1000^n.


#2

Also, in response to the rather-cool user-poll above (indeed, administrators of http://discussion.DreamHost.com , thanks for allowing ordinary users to create polls!),

For my answers, for me “adding a new file over .5GB”:[list][]“fails currently without doing anything special” (and yes I posted the file sizes & my error messages: prior post)
[
]“succeeded in the past (over 1mo) without doing anything special” (and yes I posted the 1.03GB file size: prior post)[/list]
But the when I click the “Vote!” button, the voting doesn’t work for me (does it for you?) so:[list=1]
[]I’m posting my vote here so everyone can know it (it is as you would infer it from my prior post)
[
]Especially for administrators of this http://discussion.DreamHost.com , I’m alerting that the website doesn’t allow voting in this poll (at least for me) and inquiring why:[list=a]
[]What happened was the “Vote!” button just sent to URL http://discussion.dreamhost.com/polls.php?action=vote&pid=28&option[2]=1&option[4]=1 which displayed a completely blank page, and when I clicked Show Results I see my results didn’t count (so tragically there is not even 1 vote cast!). I tried casting my vote both in latest Firefox & Chrome, both logged in & not-logged in, all with this same result.
[
]Does this polling mechanism work for anyone, or anyone else experience this?
[]Is the problem that: the author of the poll (me) can’t vote in the poll (why? That would seem wrong) but this discussion software doesn’t report any warning of this just a confusing blank page when the poll creator attempts to vote? Or what is the problem?
[
]What are the plans to fix this? --especially since this seems it would be rather cool if it worked.[/list]Thanks[/list]


#3

As far as polls go, I was planning a while back to use them for a suggestions system, but that’s not currently happening. The issues you’re running into with your poll are a result of that… I’ve fixed things (sort of) by just disabling polls for now.

As far as your question goes — Subversion is far from ideal for storing VMDKs. It’s primarily designed for storing relatively small files, such as source code; it doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly). You’re likely to be much better off just uploading the image and serving it over HTTP, possibly in chunks.


#4

(post under construction, just saved now for safety) Dear andrewf,

Thanks for your your reply! :slight_smile:

Thanks for heeding my concern on Polls. [list=1]
[]Before having all polls disappear, did you you save the polls out there? Fortunately after posting the Q I saved it all (to here) including the poll that I created (important if you/Dreamhost ever get polls working again), but other folks might not have.
[
]When do you expect to have polls fixed? Polls certainly aren’t urgent for me, but would be nice & helpful.[/list]
Thanks for your thoughts on Subversion for larger files. [list=1]
[]“Subversion is far from ideal for storing VMDKs” (virtual machine images) -Really? (see my point below). And if so, what would be better? As among other benefits, I want my virtual machine images to be versioned (for every time I or someone makes some changes one later discovers one wants to undo), so what else will do that well? --Especially on Dreamhost
[
]“It’s primarily designed for storing relatively small files, such as source code;” --yes, as one would guess that this and all VCS designed first for source code would have this property. But then why not extend it for larger files, too? Indeed for all files? Here is a great example: I want to version the virtual machine images for the benefits mentioned.
[] “it doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly).”. Really, how do you know? As actually so far my experience here as been that Subversion DOES handle them (large binary files being diffed), and remarkably well. Take my most recent Subversion commit of 1.4GiB AppData\Local\Microsoft\Outlook\ (operation LQB2C6), which consists of a 1.2GiB .ost and .2GiB .pst file (which are both binary, especially the encrypted .ost). for all all VCS used for source code: based on the diffs (which were then binary), this was a mere 127MiB, so just 8.7%, and a mere 24 minutes to commit a new VERSION of 1.4GiB! --I call that excellent binary diff (compact, handling a very large files, and fast), especially that it’s diffing over a compressible-encrypted binary file of unknown format! But now it won’t work at all (for any new files, or for anybody else now trying to do it) now that DreamHost has apparently quietly inserted this severe .5GB file/checkin size limit, a key reason for my post here.
[
]“it doesn’t handle large files well” --Rather, in my 1st post I cite official references saying that Subversion CAN handle files of of arbitrary sizes (and, as mentioned, have experienced for a 1.03GB file, on Dreamhost) so again it seems the problem is some change on Dreamhost, so I’d like to get that answered. Yes, because Subversion doesn’t resume after partial transfers, it will suffer problems if one doesn’t have a reliable Internet connection (and can wait to do the full transfer), but I have one and can do the wait, and many folks also can, so I had no problem in that original transfer of 1.03GB and expect I could go to 10GB or perhaps larger without a hitch.
[]Again, it’s easy to say “use something else” but What specific alternatives are there? If you know of something better than Subversion for for online large file storage --which which will also version files (ideally as nicely), handle private storage, and ideally work on Dreamhost (just $.10/GB or less) or be similarly inexpensive-- then please post it here.[list=1]
[
]http://en.wikipedia.org/wiki/Google_Docs provides personal cloud storage remarkably inexpensive for personal use ($.025/GiB/mo), and saves arbitrary versions (though suffers with that anyone with write access can delete any of them), and now allows any file up to 10GB, but most seriously I’ve yet to clearly find “GDrive” (drive mount) software which works on (all top 3 platforms=Windows+Mac+Linux), though http://mezeo.com/mezeo-ready-cloud-gateway-solution-partners looks like it may lead to some.
[]http://en.wikipedia.org/wiki/Wuala seems would seem an option (practically free storage, and 14GB file size limit) but that doesn’t run on DreamHost and has a notable limit (“up to 100 GB per computer”) on the inexpensive storage and I am not confident it does versions well: “only 10 versions will be saved” and I would doubt it can keep permanent references to prior versions without allocating the space again (as making a copy).[/list]
[
]In the meantime, yes I’d like to get this Q answered (on what’s wrong on on Dreamhost Subversion, and how to fix it).[/list]


#5

(post under construction, just saved now for safety) Dear andrewf,

Thanks for your your reply! :slight_smile:

I see you are able to to administer the polls on this site http://discussion.dreamhost.com --cool! For my polls Q, it seems I couldn’t have reached someone better. So then what is your DreamHost team profile? I couldn’t find you on http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles . So please put your DreamHost team profile link on your discussion profile. Thanks.

Thanks for heeding my concern on the polling mechanism here. [list=1]
[]Yep, I agree to turn off the polling mechanism if not yet working; will safe future posters this frustration I went thru.
[
]Before having all polls disappear, did you you save the polls out there? Fortunately after posting the Q I saved it all (to here) including the poll that I created (important if you/Dreamhost ever get polls working again), but other folks might not have.
[]When do you expect to have polls fixed? Polls certainly aren’t urgent for me, but would be nice & helpful.[/list]
Thanks for your thoughts on Subversion for larger files. [list=1]
[
]You write “Subversion is far from ideal for storing VMDKs” (virtual machine images) -Really? Except for the recent bug I post here apparently introduced by DreamHost, that hasn’t been my understanding & nor experience, as I detail next. But if it is true, what would be better? As among other benefits, I want my virtual machine images to be versioned (for every time I or someone makes some changes one later discovers one wants to undo), so what else will do that well? --Especially on Dreamhost (more on that below)
[]You write Subversion is “primarily designed for storing relatively small files, such as source code;” --yes, as one would guess that this and all VCS designed first for source code would have this property. But then why not extend it for larger files, too? Indeed for all files? And here is a great example: I want to version the virtual machine images for the benefits mentioned.
[
]You write Subversion “doesn’t handle large files well” --Rather, in my 1st post I cite official references saying that Subversion CAN handle files of arbitrary sizes, and, until Dreamhost apparently introduced this bug, that indeed has been my experience: take that example I gave of a 1.03GB file sent to Dreamhost Subversion without a hitch, so again it seems the problem is some bug introduced on Dreamhost, so I’d like to get that answered.
Yes, you’re right if you are suggesting the problem that Subversion doesn’t resume after partial transfers (unlike BitTorrent and many other lesser file transfer software), so it will fail IF if a file (or commit) is big AND (one doesn’t have a reliable Internet connection or can’t wait to do the full transfer); but I have a reliable Internet connection and can certainly wait for a transfer as I’ve also got a fast (broadband) connection >1Mbps so the wait isn’t much --and many folks are also in my shoes (arguably anyone with a decent broadband Internet link in home or office), so I had no problem in that original transfer of 1.03GB and expect I could go to 10GB or perhaps larger without a hitch, as would seemingly be the case for millions of other people. So this most notable drawback of Subversion for large files (& commits) (not able to do a partial checkin) is not a problem for me nor seemingly the millions with solid broadband Internet; but this, Dreamhost apparently suddenly having a ~.5GB checkin/file limit, IS of course a problem.
[]You write Subversion “doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly).”. Really, how do you know? As actually so far my experience here as been that Subversion DOES handle them (large binary files being diffed), and remarkably well. Take my most recent Subversion commit of 1.4GiB AppData\Local\Microsoft\Outlook\ (operation LQB2C6), which consists of a 1.2GiB .ost and .2GiB .pst file (which are both binary, especially the encrypted .ost). for all all VCS used for source code: based on the diffs (which were then binary), this was a mere 127MiB, so just 8.7%, and a mere 24 minutes to commit a new VERSION of 1.4GiB! --I call that excellent binary diff (compact, handling a very large files, and fast), especially that it’s diffing over a compressible-encrypted binary file of unknown format! And this performance is typical there. But now it won’t work at all (for any new files, or for anybody else now trying to do it) now that DreamHost has apparently quietly inserted this severe .5GB file/checkin size limit, a key reason for my post here.
[
]It’s easy to blame it on Subversion, but then what specifically is the problem with Subversion? I don’t find anything, but I do of course find a problem if apparently-DreamHost introduces a .5GB size limit. So still I’d like to get this Q answered (on what’s wrong on on Dreamhost Subversion, and how to fix it).
[]But say you were right, say Subversion “doesn’t handle large files well”, or say, seemingly more likely, that DreamHost won’t fix this scalability bug now introduced in their Subversion. Then, specifically, what alternatives are there? If you know of something better than Subversion for for online large file storage --which which will also version files (ideally as nicely), handle private storage, and ideally work on Dreamhost (just $.10/GB or less) or be similarly inexpensive-- then please post it here.[list=1]
[
]http://en.wikipedia.org/wiki/Google_Docs provides personal cloud storage remarkably inexpensive for personal use ($.025/GiB/mo), and saves arbitrary versions (though suffers with that anyone with write access can delete any of them), and now allows any file up to 10GB, but most seriously I’ve yet to clearly find “GDrive” (drive mount) software which works on (all top 3 platforms=Windows+Mac+Linux), though http://mezeo.com/mezeo-ready-cloud-gateway-solution-partners looks like it may lead to some.
[*]http://en.wikipedia.org/wiki/Wuala seems would seem an option (practically free storage, and 14GB file size limit) but that doesn’t run on DreamHost and has a notable limit (“up to 100 GB per computer”) on the inexpensive storage and I am not confident it does versions well: “only 10 versions will be saved” and I would doubt it can keep permanent references to prior versions without allocating the space again (as making a copy).[/list][/list]


#6

Dear andrewf,

Thanks for your reply! :slight_smile:

I see you are able to to administer the polls on this site http://discussion.dreamhost.com --cool! For my polls Q, it seems I couldn’t have reached someone better. So then what is your DreamHost team profile? I couldn’t find you on http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles . Please put your DreamHost team profile link on your discussion profile. Thanks.

Thanks for heeding my concern on the polling mechanism on this site. [list=1]
[]Yep, I agree to turn off the polling mechanism if not yet working; will save future posters this small frustration I went thru (of creating it but it wouldn’t run).
[
]Before having all polls disappear, did you you save the polls out there? Fortunately right after posting my Q & poll I saved it all (to here), important if you/Dreamhost ever get polls working again; but other folks probably didn’t go thru that trouble. Did they loose their polls so they won’t have to recreate them when it’s working, or if they want to recreate them then somewhere else?
[]When do you expect to have polls fixed? Polls certainly aren’t urgent for me, but would be nice & helpful.[/list]
Thanks for your thoughts on Subversion for larger files. [list=1]
[
] So you astutely notice that I’m using Subversion for “storing VMDKs” (virtual machine images) – nice! And, yes, as surprising as that might sound, it’s by no accident, as read next on what Subversion can & does do. Indeed so far it seems the only problem is DreamHost, NOT Subversion, as I detail again next.
[]You write “Subversion is far from ideal for storing VMDKs” (virtual machine images) -Really? As, except for the recent bug I post here apparently introduced by DreamHost, that hasn’t been my understanding & nor experience, as I detail next. But if it is true, then what would be better? (see my last point) For, among other benefits Subversion provides, I want my virtual machine images to be versioned (for every time I or someone makes some changes one later discovers one wants to undo), and via only storing the (yes, compact) differences between versions; so what else will do that well? --Especially on Dreamhost. For instance, do the virtual machines have built-in version maintenance? And is it free? And (hard to believe) as powerful & flexible as Subversion or other VCS?
[
]You write Subversion is “primarily designed for storing relatively small files, such as source code;” --yes, as one would guess that this and all VCS designed first for source code would have this property. But then why not extend it for larger files, too? Indeed for all files? And here is a great example: I want to version the virtual machine images for the benefits mentioned.
[]You write Subversion “doesn’t handle large files well” --Rather, in my 1st post I cite official references saying that Subversion CAN handle files of arbitrary sizes, and, until Dreamhost apparently introduced this bug, that indeed has been my experience: take that example I gave of a 1.03GB file sent to Dreamhost Subversion without a hitch, so again it seems the problem is some bug introduced on Dreamhost, so I’d like to get that answered.
Yes, you’re right if you are suggesting the problem that Subversion doesn’t resume after partial transfers (unlike BitTorrent and many other lesser file transfer software), so it will fail IF if a file (or commit) is big AND (one doesn’t have a reliable Internet connection or can’t wait to do the full transfer); but I have a reliable Internet connection and can certainly wait for a transfer as I’ve also got a fast (broadband) connection >1Mbps so the wait isn’t much --and many folks are also in my shoes (arguably anyone with a decent broadband Internet link in home or office), so I had no problem in that original transfer of 1.03GB and expect I could go to 10GB or perhaps larger without a hitch, as would seemingly be the case for millions of other people. So this most notable drawback of Subversion for large files (& commits) (not able to do a partial checkin) is not a problem for me nor seemingly the millions with solid broadband Internet; but this, Dreamhost apparently suddenly having a ~.5GB checkin/file limit, IS of course a problem.
[
]You write Subversion “doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly).”. Really, how do you know? As actually so far my experience here as been that Subversion DOES handle them (large binary files being diffed), and remarkably well. Take my most recent Subversion commit of 1.4GiB AppData\Local\Microsoft\Outlook\ (operation LQB2C6), which consists of a 1.2GiB .ost and .2GiB .pst file (which are both binary, especially the encrypted .ost). for all all VCS used for source code: based on the diffs (which were then binary), this was a mere 127MiB, so just 8.7%, and a mere 24 minutes to commit a new VERSION of 1.4GiB! --I call that excellent binary diff (compact, handling a very large files, and fast), especially that it’s diffing over a compressible-encrypted binary file of unknown format! And this performance is typical there. But now it won’t work at all (for any new files, or for anybody else now trying to do it) now that DreamHost has apparently quietly inserted this severe .5GB file/checkin size limit, a key reason for my post here.
[]It’s easy to blame it on Subversion, but then what specifically is the problem with Subversion? I don’t find anything, but I do of course find a problem if apparently-DreamHost introduces a .5GB size limit. So still I’d like to get this Q answered (on what’s wrong on on Dreamhost Subversion, and how to fix it).
[
]But say you were right, say Subversion “doesn’t handle large files well”, or say, seemingly more likely, that DreamHost won’t fix this scalability bug now introduced in their Subversion. Then, specifically, what alternatives are there? If you know of something better than Subversion for for online large file storage --which which will also version files (ideally as nicely), handle private storage, and ideally work on Dreamhost (just $.10/GB or less) or be similarly inexpensive-- then please post it here.[list=1]
[]http://en.wikipedia.org/wiki/Google_Docs provides personal cloud storage remarkably inexpensive for personal use ($.025/GiB/mo), and saves arbitrary versions (though suffers with that anyone with write access can delete any of them), and now allows any file up to 10GB, but most seriously I’ve yet to clearly find “GDrive” (drive mount) software which works on (all top 3 platforms=Windows+Mac+Linux), though http://mezeo.com/mezeo-ready-cloud-gateway-solution-partners looks like it may lead to some.
[
]http://en.wikipedia.org/wiki/Wuala seems would seem an option (practically free storage, and 14GB file size limit) but that doesn’t run on DreamHost and has a notable limit (“up to 100 GB per computer”) on the inexpensive storage and I am not confident it does versions well: “only 10 versions will be saved” and I would doubt it can keep permanent references to prior versions without allocating the space again (as making a copy).[/list][/list]


#7

Dear andrewf,

Thanks for your reply :slight_smile:

Thanks for heeding my concern on the polling mechanism on this site. [list=1]
[]I see you are able to to administer the polls on this site http://discussion.dreamhost.com --cool! For my polls Q, it seems I couldn’t have reached someone better. So then what is your DreamHost team profile? I couldn’t find you on http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles . Please put your DreamHost team profile link on your discussion profile. Thanks.
[
]Yep, I agree to turn off the polling mechanism if not yet working; will save future posters this small frustration I went thru (of creating it but it wouldn’t run).
[]Before having all polls disappear, did you you save the polls out there? Fortunately right after posting my Q & poll I saved it all (to here), important if you/DreamHost ever get polls working again; but other folks probably didn’t go thru that trouble. Did they loose their polls so they won’t have to recreate them when it’s working, or if they want to recreate them then somewhere else?
[
]When do you expect to have polls fixed? Polls certainly aren’t urgent for me, but would be nice & helpful.[/list]
Thanks for your thoughts on Subversion for larger files. [b]The gist of my reply is: actually, though it may be hard to believe, as my original post suggested:
*Subversion’s been & seems great (yes, even for big binary files).

  • Instead, DreamHost seems the problem here --so what to do? What fixes? And what alternatives, esp. if DreamHost doesn’t this bug apparently with their Subversion?[/b][list=1]
    [] So you astutely notice that I’m using Subversion for “storing VMDKs” (virtual machine images) – nice! And, yes, as surprising as that might sound, it’s by no accident, as read next on what Subversion can & does do. Indeed so far it seems the only problem is DreamHost, NOT Subversion, as I detail again next.
    [
    ]You write “Subversion is far from ideal for storing VMDKs” (virtual machine images) -Really? As, except for the recent bug I post here apparently introduced by DreamHost, that hasn’t been my understanding & nor experience, as I detail next. But if it is true, then what would be better? (see my last point) For, among other benefits Subversion provides, I want my virtual machine images to be versioned (for every time I or someone makes some changes one later discovers one wants to undo), and via only storing the (yes, compact) differences between versions; so what else will do that well? --Especially on Dreamhost. For instance, do the virtual machines have built-in version maintenance? And is it free? And (hard to believe) as powerful & flexible as Subversion or other VCS?
    []You write Subversion is “primarily designed for storing relatively small files, such as source code;” --yes, as one would guess that this and all VCS designed first for source code would have this property. But then why not extend it for larger files, too? Indeed for all files? And here is a great example: I want to version the virtual machine images for the benefits mentioned.
    [
    ]You write Subversion “doesn’t handle large files well” --Rather, in my 1st post I cite official references saying that Subversion CAN handle files of arbitrary sizes, and, until Dreamhost apparently introduced this bug, that indeed has been my experience: take that example I gave of a 1.03GB file sent to Dreamhost Subversion without a hitch, so again it seems the problem is some bug introduced on Dreamhost, so I’d like to get that answered.
    Yes, you’re right if you are suggesting the problem that Subversion doesn’t resume after partial transfers (unlike BitTorrent and many other lesser file transfer software), so it will fail IF if a file (or commit) is big AND (one doesn’t have a reliable Internet connection or can’t wait to do the full transfer); but I have a reliable Internet connection and can certainly wait for a transfer as I’ve also got a fast (broadband) connection >1Mbps so the wait isn’t much --and many folks are also in my shoes (arguably anyone with a decent broadband Internet link in home or office), so I had no problem in that original transfer of 1.03GB and expect I could go to 10GB or perhaps larger without a hitch, as would seemingly be the case for millions of other people. So this most notable drawback of Subversion for large files (& commits) (not able to do a partial checkin) is not a problem for me nor seemingly the millions with solid broadband Internet; but this, DreamHost apparently suddenly having a ~.5GB checkin/file limit, IS of course a problem.
    []You write Subversion “doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly).”. Really, how do you know? As actually so far my experience here as been that Subversion DOES handle them (large binary files being diffed), and remarkably well. Take my most recent Subversion commit of 1.4GiB AppData\Local\Microsoft\Outlook\ (operation LQB2C6), which consists of a 1.2GiB .ost and .2GiB .pst file (which are both binary, especially the encrypted .ost). for all all VCS used for source code: based on the diffs (which were then binary), this was a mere 127MiB, so just 8.7%, and a mere 24 minutes to commit a new VERSION of 1.4GiB! --I call that excellent binary diff (compact, handling a very large files, and fast), especially that it’s diffing over a compressible-encrypted binary file of unknown format! And this performance is typical there. But now it won’t work at all (for any new files, or for anybody else now trying to do it) now that DreamHost has apparently quietly inserted this severe .5GB file/checkin size limit, a key reason for my post here.
    [
    ]It’s easy to blame it on Subversion, but then what specifically is the problem with Subversion? I don’t find anything, but I do of course find a problem if apparently-DreamHost introduces a .5GB size limit. So still I’d like to get this Q answered (on what’s wrong on on Dreamhost Subversion, and how to fix it).
    []But say you were right, say Subversion “doesn’t handle large files well”, or say, seemingly more likely, that DreamHost won’t fix this scalability bug now introduced in their Subversion. Then, specifically, what alternatives are there? If you know of something better than Subversion for for online large file storage --which which will also version files (ideally as nicely), handle private storage, and ideally work on Dreamhost (just $.10/GB or less) or be similarly inexpensive-- then please post it here.[list=1]
    [
    ]http://en.wikipedia.org/wiki/Google_Docs provides personal cloud storage remarkably inexpensive for personal use ($.025/GiB/mo), and saves arbitrary versions (though suffers with that anyone with write access can delete any of them), and now allows any file up to 10GB, but most seriously I’ve yet to clearly find “GDrive” (drive mount) software which works on (all top 3 platforms=Windows+Mac+Linux), though http://mezeo.com/mezeo-ready-cloud-gateway-solution-partners looks like it may lead to some.
    []http://en.wikipedia.org/wiki/Wuala seems would seem an option (practically free storage, and 14GB file size limit) but that doesn’t run on DreamHost and has a notable limit (“up to 100 GB per computer”) on the inexpensive storage and I am not confident it does versions well: “only 10 versions will be saved” and I would doubt it can keep permanent references to prior versions without allocating the space again (as making a copy).
    [
    ]What others?[/list][/list]

#8

Dear andrewf,

Thanks for your reply :slight_smile:

Thanks for heeding my concern on the polling mechanism on this site http://discussion.dreamhost.com .[list=1]
[]I see you are able to administer the polls on this site --cool! For my polls Q, it seems I couldn’t have reached someone better. So then what is your DreamHost team profile? I couldn’t find you on http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles . Please put your DreamHost team profile link on your discussion profile. Thanks.
[
]Yep, I agree to turn off the polling mechanism if not yet working; will save future posters this small frustration I went thru (of creating it but it wouldn’t run).
[]Before having all polls disappear, did you save the polls out there? Fortunately right after posting my Q & poll I saved it all (to here), important if you/DreamHost ever get polls working again; but other folks probably didn’t go thru that trouble. Did they loose their polls so they won’t have to recreate them when it’s working, or if they want to recreate them then somewhere else?
[
]When do you expect to have polls fixed? Polls aren’t urgent for me, but would be nice & helpful.[/list]
Thanks for your thoughts on Subversion for larger files. [b]The gist of my reply is: though, yes, it may be hard to believe, actually & just as my original post suggested:
*Subversion’s been & seems great (yes, even for big (>1GB) binary files of unknown format: fast-diffed to 8.7% --seeming remarkably good performance!).

  • Instead, DreamHost seems the problem here [/b](this bug which just appeared in DreamHost’s Subversion (wasn’t there 2 months ago), and I haven’t heard of it anywhere else and it wouldn’t seem to be anywhere else including that Subversion says Subversion has no practical size limits (cited in 1st post) and “.5GB” seems like some # some admin picked (which then would be a DreamHost admin) including “.5GB” does not seem to occur in any of the raw software)–so what to do? What fixes? And what alternatives, esp. if DreamHost doesn’t/won’t fix this bug apparently with their Subversion?[list=1]
    [] So you astutely notice that I’m using Subversion for “storing VMDKs” (virtual machine images) – nice! And, yes, as surprising as that might sound, it’s by no accident, as read next on what Subversion can & does do. Indeed so far it seems the only problem is DreamHost, NOT Subversion, as I detail again next.
    [
    ]You write “Subversion is far from ideal for storing VMDKs” (virtual machine images) -Really? As, except for the recent bug I post here apparently introduced by DreamHost, that hasn’t been my understanding & nor experience, as I detail next. But if it is true, then what would be better? (see my last point) For, among other benefits Subversion provides, I want my virtual machine images to be versioned (for every time I or someone makes some changes one later discovers one wants to undo), and via only storing the (yes, compact) differences between versions; so what else will do that well? --Especially on DreamHost. For instance, do the virtual machines have built-in version maintenance? And is it free? And (hard to believe) as powerful & flexible as Subversion or other VCS?
    []You write Subversion is “primarily designed for storing relatively small files, such as source code;” --yes, as one would guess that this and all VCS designed first for source code would have this property. But then why not extend it for larger files, too? Indeed for all files? And here is a great example: I want to version the virtual machine images for the benefits mentioned.
    [
    ]You write Subversion “doesn’t handle large files well” --Rather, in my 1st post I cite official references saying that Subversion CAN handle files of arbitrary sizes, and, until DreamHost apparently introduced this bug, that indeed has been my experience: take that example I gave of a 1.03GB file sent to DreamHost Subversion without a hitch, so again it seems the problem is some bug introduced on DreamHost, so I’d like to get that answered.
    Yes, you’re right if you are suggesting the problem that Subversion doesn’t resume after partial transfers (unlike BitTorrent and many other lesser file transfer software), so it will fail IF a file (or commit) is big AND (one doesn’t have a reliable Internet connection or can’t wait to do the full transfer); but I have a reliable Internet connection and can certainly wait for a transfer as I’ve also got a fast (broadband) connection >1Mbps so the wait isn’t much --and many folks are also in my shoes (arguably anyone with a decent broadband Internet link in home or office), so I had no problem in that original transfer of 1.03GB and expect I could go to 10GB or perhaps larger without a hitch, as would seemingly be the case for millions of other people. So this most notable drawback of Subversion for large files (& commits) (not able to do a partial checkin) is not a problem for me nor seemingly the millions with solid broadband Internet; but this, DreamHost apparently suddenly having a ~.5GB checkin/file limit, IS of course a problem.
    []You write Subversion “doesn’t handle large files well, especially when they’re binary files (which it can’t “diff” properly).”. [list=1]
    [
    ]Really, how do you know? As actually
    []so far my experience here as been that Subversion DOES handle them (large binary files being diffed), and remarkably well. Take my most recent Subversion commit of 1.4GiB AppData\Local\Microsoft\Outlook\ (operation LQB2C6), which consists of a 1.2GiB .ost and .2GiB .pst file (which are both binary, especially the encrypted .ost). for all VCS used for source code: based on the diffs (which were then binary), this was a mere 127MiB, so just 8.7%, and a mere 24 minutes to commit a new VERSION of 1.4GiB! --I call that excellent binary diff (compact, handling a very large files, and fast), especially that it’s diffing an unknown binary format.
    [
    ](And was the .ost also compressible-encrypted as it used to be (making the diff job even tougher, probably much tougher)? I think not, as that option is no longer listed (in my Outlook 2007), indeed as it no longer seems to be needed as encryption seems to have been removed in Outlook 2007 in favor of the (far more comprehensive) filesystem encryption.)
    []And this great performance (on big binary files) is typical on all the .pst & .ost files I’ve tested on, which have been all the big binary files I’ve encountered, until now when I also want to do & try it on virtual machine images.
    [
    ]But now on DreamHost it won’t work at all (for seemingly-any type of new file, including seemingly for anybody else now trying to do it) now that DreamHost has apparently quietly inserted this severe .5GB file/checkin size limit, my #1 reason for posting here.[/list]
    []Yes, I could imagine folks casually blaming Subversion off-hand, because often http://en.wikipedia.org/wiki/Version_Control_System s Version Control Systems are foremost for software source code (as just as Subversion is) and so weren’t designed for so can’t handle big and/or binary files. BUT Subversion seems fine here (except for partial commits, which is easy to overcome now that good Internet connections are common). So then what specifically is the problem with Subversion? I don’t find anything, but I and anyone would find a problem for big files/commits if there is suddenly a ~.5GB size limit bug, which DreamHost has apparently introduced. So still I’d like to get this Q answered (on what’s wrong on DreamHost Subversion, and how to fix it).
    [
    ]But say you were right, say Subversion “doesn’t handle large files well”, or say, seemingly more likely, that DreamHost won’t fix this scalability bug now introduced in their Subversion. Then, specifically, what alternatives are there? If you know of something better than Subversion for online large file storage --which will also version files (ideally as nicely), handle private storage, and ideally work on DreamHost (just $.10/GB or less) or be similarly inexpensive-- then please post it here.[list=1]
    []My current 1st choice: http://en.wikipedia.org/wiki/Google_Docs provides personal cloud storage remarkably inexpensive for personal use ($.025/GiB/mo), and saves arbitrary versions (though suffers with that anyone with write access can delete any of them), and now allows any file up to 10GB, but most seriously I’ve yet to clearly find “GDrive” (drive mount) software which works on (all top 3 platforms=Windows+Mac+Linux), though http://mezeo.com/mezeo-ready-cloud-gateway-solution-partners looks like it may lead to some.
    [
    ]My current 2nd choice: http://en.wikipedia.org/wiki/Wuala seems would seem an option (practically free storage, and 14GB file size limit) but that doesn’t run on DreamHost and has a notable limit (“up to 100 GB per computer”) on the inexpensive storage and I am not confident it does versions well: “only 10 versions will be saved” and I would doubt it can keep permanent references to prior versions without allocating the space again (as making a copy).
    [*]Any others?[/list][/list]

#9

Ok, it seems no readers of this board know the answer, which is not surprising since, although this problem could seemingly affect anyone, it seems an internal DreamHost problem but I imagine in these user forums are read mostly by just ordinary users);
so I will now bring this issue to the attention of DreamHost Support:[list=1]
[]Sent: now (“Aug 30, 2011 at 4:57 PM”), “tracking # is 4474649”.
[
]form URL: https://panel.dreamhost.com/index.cgi?tree=support.msg& (Contact Support) choosing “Other” -> “Subversion”
[]Subject: My DreamHost Subversion now can’t add any new file over ~.5GB
[
]Message: See http://discussion.dreamhost.com/thread-131024-post-140952.html#pid140952 (link to this post)

Dear DreamHost Support, especially Subversion specialists,

I am experiencing a new bug, apparently coming from some changed DreamHost software, which is calling a halt on all my work in Subversion with larger files.
And the new bug is serious:[list=1]
[]if not solved, sadly it may cause me to leave DreamHost Subversion (it now-especially not actually being a comprehensive solution, even though Subversion seems capable of that), and maybe eventually even have me leave DreamHost altogether, as DreamHost’s best price in $/GB for private Subversion storage was (and still is) the #1 reason which sold me DreamHost, and was mostly working (except for lack of indexing & fine-permission control, which may be fixable) but now suddenly is NOT working for a number of significant files.
[
]And, from what I can tell thus far, this new bug would affect not just me, but potentially all DreamHost users.[/list]
The whole problem & discussion is posted here, on a thread of the DreamHost user discussion forums:[list=1]
[]Post 1 seemingly-fully details the bug & situation
[
]Post 3 touches on a common objection to this use: that Subversion “won’t” work well for large files, especially binary files
[]Post 4 responds with[list=1][]a full explanation of why in fact Subversion should work & work well for large files & even large binary files. and actually does & DID before here on DreamHost, until this bug appeared and put an end to all new such files, plus
[]asks specifically what else would provide the same capabilities, listing 2 alternatives, but all those solutions involve instead NOT using DreamHost[/list][/list]
So please post your thoughts and response & solutions here. Thanks.
.
[
]Relevant URL: gave URL of particular Subversion repository (confidential to DreamHost support so not posted on this public site)
[]Has it ever worked before? YES
[
]type of this request: “I can’t get things done until I hear back from you, please reply ASAP”
[]your general expertise in the area of this request: “I have a good understanding of this stuff”
[
]Phone Callback / Live Chat: No, please post here, especially so all users can benefit, especially as this seems it could affect any user[/list]


#10

To this [quote=“CaydenCM, post:9, topic:55881”]…
bring this issue to the attention of DreamHost Support:…[/quote], DreamHost (too) promptly replies:[quote]
From: DreamHost Customer Support Team support@dreamhost.com
Date: Tue, Aug 30, 2011 at 6:34 PM
Subject: Re: [… 49209965] My DreamHost Subversion now can’t add any new file over ~.5GB
To: Me

  • After reading this response, please consider visiting the survey below to comment on its quality. Thanks!
    http://www.dreamhost.com/survey.cgi?n=49209965&m=9965020
  • If the service you received from us was exceptional, please consider tweeting your love for @dreamhost. It’ll warm our hearts, soothe our souls, and get you good karma at some point down the road.

Hey there,

Although Subversion will try to painstakingly chug through committing several-hundred-megabyte files, that’s really not something that Subversion handles well. If it works at all, it will take quite a while, and as you work on the repository it’s going to become slower and slower to access.

I would generally recommend storing larger files outside of Subversion for this reason. If you need to pair up particular large files with a particular revision in a Subversion repository, then you might try including revision information in the filenames of the large files and including a text file in Subversion that indicates which file revision goes with that repository revision.

If you need anything else, please let me know!

Thanks,
Benjamin R[/quote]

Benjamin, to be frank & in a nutshell:[list=1]
[]Your reply (mis)suggests you’ve helped & even solved the issue, even concluding “If you need anything else, please let me know!” (as if the issue is now closed)
when actually your reply hasn’t even answered the main Q (how to undo this new bug in DreamHost’s Subversion), indeed not even addressed this the key issue (potentially even suggesting of your boss DreamHost that DreamHost wouldn’t want to admit & face if had a new bug, something doubt they would appreciate)
[
]and your reply makes claims which are vague (and incorrect), and without citing any sources for them, by saying what I’m asking to do won’t work (“painstakingly chug … If it works at all”)
most ironically when actually I just reported to you here, repeatedly, that not only can it work but did, repeatedly & beautifully (that is of course until this new bug at DreamHost), all of which one could essentially infer even from just reading the subject line of this thread & the bug report (“My DreamHost Subversion now can’t add any new file over ~.5GiB or .5GB”),
and also in contradiction to your claims, this is inline the official (Subversion) docs say, which I cite
[] but you don’t say word to any of this (how your response completely contradicts these facts presented to you), so you did you even bother reading the bug report you’re replying to??
[
]And you suggest an alternative solution that suggests the person posing this issue is dumb (as clearly, if s/he was using Subversion and could file a bug report as this, s/he would have thought of storing say large files outside of Subversion and could almost-certainly handle the problem of “you need to pair up particular large files with a particular revision in a Subversion repository”); and the alternative you suggest seems to suggest (probably accurately) that you are missing the very point (which was explained to you, with examples) that if the person is using Subversion for large files (as opposed to say regular WebDAV, or SFTP, or local storage), then it is because the person needs the additional features which Subversion offers (as versioning of large files, with compact binary differencing & upload) so that’s the big issue, not the much easier task of pairing up files between storage locations! Indeed, the bug report already says “this is my Subversion repository for oversized items & collections” so the person already is pairing up files.
[*]And completely ignoring my 2x request "“please post here”[/list] So instead of helping & solving the issue as your reply suggests (indeed, most arrogant of at all, even suggesting you’ve solved the key issue(!), when in fact not even addressing it, not even asking if you have solved it), instead your reply seemingly just wastes everyone’s time and angers me --with its over-confidence & technical cavalierness. Please don’t do that disrespect. Again this is a serious issue. So please read this thread carefully, and answer its Qs (especially how to undo this new bug on DreamHost); else have someone else do it who can.

Now if you gave these same answers you did with a point of view of humility & seeming-honesty, as “I don’t really know for sure and I haven’t really read the bug report, but here are some passing thoughts I have”, that would be okay (well except for probably not reading the bug report) --that is IFF you were just an ordinary user. But you are official DreamHost Support. So then, if you don’t know it or can’t handle it, you are expected to get a professional who can & will, and also learn from him/her & what they do if doing that is supposed to be your job, too.


actually that (which also contradict all the official sources & REAL EXPERIENCE I just cited to you, but you don’t say a word to this – did you even bother reading it??)

  • and suggest an alternative

apparently because you seemingly didn’t carefully read this the bug report, nor really define & substantiate your claims ,

Now point-by-point, substantiating this:[list=1][]I posted for you above “please post your thoughts and response & solutions here” and “please post here, especially so…”. But instead you just directly emailed me, and with not a word why you did differently. Why did you do that? As naturally then it causes me to have to repost your response (so everyone else can see it, and plus to keep all the discussion together). Did you not see what I wrote for you? I listed it 2x, including once in bold. Please just do that simple request, else explain why you cannot (that is, when I and DreamHost users are expected to post here and can do so just fine, yet somehow this is truly too hard for a DreamHost staff member).
[
]As if you’ve solved the issue, you flippantly conclude “If you need anything else”!? Of course one would need something else! Because you didn’t answer the # question (how undo this new bug), nor even address it! Instead you’ve tried to convince me that what I want to do won’t work when, quite ironically, you seem to be missing that I am reporting to you right here that I’ve seen & experienced it work, and work beautifully, and right here on DreamHost! So did you even carefully read this thread? For yes, I wrote repeatedly, big files have worked beautifully, that is, until this bug, so please, how do we get this new bug undone?, especially when it seems to be from something DreamHost did.
[]But say it was true what you claim, that “Subversion will try to painstakingly chug through committing several-hundred-megabyte files, that’s really not something that Subversion handles well. If it works at all”. Then of course how would you explain what I just reported to you that’s it’s been working, & beautifully? That, to repeat, UNTIL THIS BUG, all my experience is, DreamHost Subversion handled beautifully all the big files I’ve tested it on, all the way up to my 1.2GiB .ost file, including fast & compact (8.7%) binary diffs. So either you’re not reading (my guess) or you’re telling me my measurements are false, or I’m lying. So if in fact what you’re saying is true, “if [uploading] works at all”, then, of course, how do you refute that, until this bug, it’s been working no-problem on DreamHost? And, not surprisingly, because, again, the official Subversion cite it as being scalable: “A nice feature of Subversion is that by design, there is no limit to the size of files it can handle. …” plus other references I cite. So how do explain that what you’re saying seems to conflict Subversion’s own docs?
[
]You also write “as you work on the repository it’s going to become slower and slower to access.” --That would be interesting IF true. But how do you know that? What source do you get that? And why then is that? For, if true, this would seem to contradict the official docs suggesting Subversion is scalable. And perhaps even more importantly, it seems to contract my real experience on DreamHost Subversion (prior to this bug), where the DreamHost Subversion repository in question, primarily for holding oversized files, has now grown to 1,616,210,482 grown now grown to And I have a hard time believing you really know that. Because when you write me “If [the upload] works at all” after I’ve just reported to you that it has BEEN working beautifully, then it sounds like you’re not reading and just guessing, wasting everyone’s time.
[]Also you write “I would generally recommend storing larger files outside of Subversion for this reason…” --okay, then how to protect against the files being accidentally/intentionally modified or deleted, as Subversion solves? And, how to compactly version them & upload the differences, Subversion solves? I need the these features of Subversion (compact differencing & differencing upload & storage, keeping ideally unlimited version, and being able to undo-else-prevent any modification or deletion, plus private storage). So how to solve that?
[
]You write “If you need to pair up particular large files with a particular revision in a Subversion repository, then you might try including revision information in the filenames” --how dumb do you think I am? Would you not think if someone uses Subversion.
[*]But most important, please get answered my key question: why is this bug now & how can it get fixed?

As this thread asks, please give specific
[So first please read this thread, Yes, it’s long, in particular because it may be hard for many people (as perhaps yourself) to believe, but it’s true.
[
[/list]


#11

Other instances of this bug happening to others, that is that I know of:[list=1]
[]One similar but 3years ago, by another DreamHost customer
[
]On non-DreamHost systems, a few other reports of this sort of failure, such as at Google Code where it was immediately fixed high-priority. The 1st post of that prior report references some of these. But from what I saw it was not clear here if the failure was always at a particular byte-quantity cut-off as the ~.5GB I’m experiencing on DreamHost, so something else may be triggering this on DreamHost.
[]No other thus far.[/list] So why do I so for only know of a few cases of this happening at DreamHost? Possibilities, in no particular order:[list=1]
[
]Perhaps I am the only one experiencing it (or one of the very few). But try it yourself (add a file over .5GiB to your DreamHost Subversion repository), and report back (on the thread) what happens in your case.
[]Perhaps others experience it but, as is common, they mis-think it’s just that Subversion can’t scale this large (since other VCS can’t) so mis-think it’s normal & shouldn’t work so don’t report it, sadly when they could be benefiting from it if only they knew it actually does work if the server is bug-free.
[
]Perhaps others are experiencing it but they only report to DreamHost directly, and DreamHost seemingly doesn’t generally share the bug reports it gets directly, except sometimes in the most general terms.
[*]Perhaps I just happen to be unfortunate one to discover & report it first. I use Subversion at DreamHost a lot & daily & for all sorts of files, so it would be not much of a surprise if I discovered it first, so it may a bit before other reports come in. But again, just try it for yourself, and report back here.[/list]


#12

Having the same issues. I’m having this issue with files over 300MB actually. It’s incredibly inconvenient and it likely would have affected my decision to sign up with Dreamhost in the first place. Never had this problem with smaller files. It’s only the bigger ones. It just seems the commit gives up for no reason at all. I’ve worked with plenty other SVN repos, using the client I’m using (and others) that do not encounter this issue.

It only seems to occur with a large file. Smaller files are fine, but the lack of flexibility to handle an occasional large file is a nuisance.


#13

I know this thread’s old, but I recently encountered this. I have a quick guess, based on the fact that I noticed people on other forums mentioning that Dreamhost shared servers would kill processes using more than 500MB of memory. I’m guessing the SVN server is either memory-mapping the file(s), or keeping the received data in memory before writing to disk.

I don’t know how to limit how much memory SVN uses, however.

.<
Mike Z


#14

I don’t think that’s it. The Subversion server process doesn’t run through suexec (i.e, it runs as the web server), so it’s not affected by our process watcher, and wouldn’t be killed for memory usage.


#15

It’s been a while since I’ve done anything with subversion. Looking at this thread I’m wondering how far I can trust it. I’m a developer working with average-sized code files, but I have dozens of projects that I want to start versioning. I’m concerned that when I do a build and create a .jar or a .exe that something in svn is going to get mucked up.

Can someone here (or anyone at DH?) confirm that this environment is rock-stable for this usage? I mean, after all, this IS the reason we’re using an environment like this in the first place, no?

Thanks.


#16

Personally I think SVN is fine for opcode. But I’m a lazy bastard who doesn’t write 1G apps :slight_smile:

I think git is the way to go if you’re into massive framework - and especially if multiple parties are concerned.


#17

I want to point out a solution to the problem in the thread title…a ghetto but workable one.

If you have a giant file to commit, you can divide it in parts that are under 500MB (I used HJSplit http://www.hjsplit.org/ ), submit the first part, append the 2nd part to it, submit the combined file which will only submit the last part, add the next part and submit, etc.

Basically, as long as the DELTA for the file is under 500MB it’ll work fine.