Discussion:
[Carbonado-interest] Volatile map repository
Brian S O'Neill
2008-05-04 20:30:39 UTC
Permalink
Carbonado now includes a volatile map repository, which is available in
the svn trunk. It will be provided in version 1.2, which I hope to
release in a few weeks.

This repository doesn't have any extra dependencies other than JDK1.6,
since it uses the ConcurrentSkipListMap class. Because the repository
does not persist anything, it is best suited for unit tests or small
in-memory caches. Check out the com.amazon.carbonado.map package to use it.

I did a quick performance comparison to BDB-JE when logging only in
memory. For simple loads by primary key, the map repository executed
over 5 times faster. Other tests which rely on more complicated query
plans tend to execute only 2 times faster than BDB-JE.

The map repository supports transactions and the isolation levels of
read committed and serializable. The locking granularity is quite
coarse, however. It behaves like a table lock. To compensate, the
repository uses upgradable locks. This allows reads to execute
concurrently, and they are only blocked by writes. Upgrades don't block
readers, but they do block writers and other upgraders.

There's a ton of new features in version 1.2, which are only summarized
in the release notes. I hope to update the documentation soon now. In
the meantime, I'll be posting more info to this list, which has been
quiet for too long.
Gunnar Wagenknecht
2008-05-05 06:15:46 UTC
Permalink
Post by Brian S O'Neill
There's a ton of new features in version 1.2, which are only summarized
in the release notes. I hope to update the documentation soon now. In
the meantime, I'll be posting more info to this list, which has been
quiet for too long.
That's great news. I think that your SVN activity is a good indicator
about the support for this project. :)

I noticed that in a branch it looked like you were also working on a
remote repository (dirmi). Are there any plans to also work on the other
repository types (i.e., propagated and partitioned)?

I was wondering about the propagated repository the other day. What kind
of propagation method would you use? JMS? I don't have any serious
experience with JMS yet but I'm facing performance concerning comments
when discussing it with colleagues. Any thoughts?

-Gunnar
--
Gunnar Wagenknecht
***@wagenknecht.org
http://wagenknecht.org/
Brian S O'Neill
2008-05-05 14:19:16 UTC
Permalink
I was getting some help for a propagated repository, but help got busy
on other projects. I created a SF project, carjam, which is to be a
lightweight JMS provider. Initially it would only support features
necessary for repository propagation. I think that a JMS provider which
uses BDB-JE for persistence will be quite fast, due to its log based format.

The persistence features are mostly complete in carjam, but actual
networking is not hooked up. Producers and consumers must be in the same
process. I have not touched the code in over a year.
Post by Gunnar Wagenknecht
Post by Brian S O'Neill
There's a ton of new features in version 1.2, which are only summarized
in the release notes. I hope to update the documentation soon now. In
the meantime, I'll be posting more info to this list, which has been
quiet for too long.
That's great news. I think that your SVN activity is a good indicator
about the support for this project. :)
I noticed that in a branch it looked like you were also working on a
remote repository (dirmi). Are there any plans to also work on the other
repository types (i.e., propagated and partitioned)?
I was wondering about the propagated repository the other day. What kind
of propagation method would you use? JMS? I don't have any serious
experience with JMS yet but I'm facing performance concerning comments
when discussing it with colleagues. Any thoughts?
-Gunnar
Gunnar Wagenknecht
2008-05-05 15:07:12 UTC
Permalink
Post by Brian S O'Neill
I was getting some help for a propagated repository, but help got busy
on other projects. I created a SF project, carjam, which is to be a
lightweight JMS provider. Initially it would only support features
necessary for repository propagation. I think that a JMS provider which
uses BDB-JE for persistence will be quite fast, due to its log based format.
ActiveMQ comes with their own persistence "logs".
http://activemq.apache.org/kaha-persistence.html

I wonder if it's a question of what consistency to provide.
http://www.allthingsdistributed.com/2007/12/eventually_consistent.html

For example, I could a imagine a specialized propagated repository which
using some kind of proprietary protocol to sync the nodes. It may use a
central server to co-ordinate synchronization or it pushes the events
out to its neighbors which in turn stream the events to their neighbors.
If such a propagated repository is combined with a replicated repository
with a big JDBC being the master "eventually consistent" might be
sufficient.

-Gunnar
--
Gunnar Wagenknecht
***@wagenknecht.org
http://wagenknecht.org/
Loading...