Current time: 04-18-2014, 11:53 PM Hello There, Guest! (LoginRegister)

report on my beta testing of Ceph so far
02-20-2011, 03:47 AM (This post was last modified: 02-20-2011 06:16 AM by pombredanne.)
Post: #1
report on my beta testing of Ceph so far
Howdy:
I am putting the ceph playground through its paces... so far so good...
ceph seems to hold up the load... there have been a few glitches on the beta server (HW or else issues) a few weeks back and the sandbox FS was wiped out clean once but it sounds stable now. This is a test environment, not a production one yet.

I have loaded in the ceph sandbox (thanks to Sage granting me an ssh directly on the sandbox) about 6TB effective content rsync'ed from open source mirrors.

df _h yields:

Code:
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sda1               30G   6.1G    22G  22% /
tmpfs                  2.1G      0   2.1G   0% /lib/init/rw
tmpfs                  2.1G      0   2.1G   0% /dev/shm
10.14.0.118,10.14.0.119,10.14.0.12:/
                        42T    13T    29T  32% /c

Funny that the ceph filesystem size is 42 teras Wink
At the moment I am using about 12TB there.
I think there is a TB or two that I did not load.

This is mostly a load test for now.
Next step will be to put theses TB of files to good use:
- for remote access to evaluate if accessing random files or lesser used files incurs some penalty and if frequently used files are served faster. And checking checksums on random known files to verify that things are not corrupted in anyway.
- for local access from a couple VPS @ dreamhost to crunch a bit through these files. For now I will do simple things like accessing *all* the files content AND filesystem metadata like computing md5 and sha1sums and directory listings... and possibly verify the computed checksums against the checksum list provided by the open source mirrors. This should check a couple things: intensive file read, and non-corruption of files on a reasonably large volume of files (aka 15 to 20 TB apparent storage and ~ 160 Millions files of varied sizes and types, including very small and some large files like Linux DVD ISOs)

Some notes:
- I am using only the file storage for now, I may be a bit old fashioned but I could not care less for S3-like object storage ... I personally do not really like the overhead of client-side access to files beyond straight HTTP for S3-like services... This comment has nothing to do with ceph nor dreamhost beta, but rather my dislike of the Amazon and S3 interface Wink
- one issue I have to solve is how to get reasonably simple access to the file storage from a VPS. Afaik FUSE (and therefore sshfs) is not available in the dreamhost kernels and I do not know (though this is likely possible) if there is a way to mount the ceph FS so that it is seen on a VPS. I will probably ending up rolling my own simple Python-based sshfs like capability so a VPS can use semi-transparently ceph storage over SSH. Any ceph team guys with ideas?

- I have the feeling that the filesystem storage overhead seems a bit high. I will know for sure when I am finished loading the puppies in ceph and will compare that with a sampling of storing the same files locally in ext4. It is possible that there are some snapshots stored automatically or other things that I do not know of .. but for now, I am under the non-confirmed fuzzy impression that storing 6TB requires 12TB.

Other than that ceph seems to rock nicely !
I will keep you posted
/HTH
Philippe
Find all posts by this user


Messages In This Thread
report on my beta testing of Ceph so far - pombredanne - 02-20-2011 03:47 AM

Forum Jump: