Questions tagged [rocksdb-java]
25 questions
3
votes
1 answer
Combine multiple Rocksdb databases
There is a use case for which I have to read huge Parquet file and convert into Rocksdb binary, So I decided to use spark (because everybody is familiar with it in my team).
And from Rocksdb side I know it's not distributed and you can not…
Kaushal
- 2,981
- 3
- 27
- 44
2
votes
0 answers
I want to make possibility creating only in-memory rocksDB in java
I have rocksDB java implementation and I want to possibility of only in-memory storing data without it writing on hdd. I think I can do this using options while creating rocksDB. I tried to make same what is writting here:…
Gleb
- 21
- 3
2
votes
0 answers
kafka streams - number of open file descriptors keeps going up
Our kafka streaming app keeps opening new file descriptors as long as they are new incoming messages without ever closing old ones. It eventually leads to exception. We've raised the limit of open fds to 65k but it doesn't seem to help.
Both Kafka…
sumek
- 24,063
- 10
- 50
- 64
1
vote
0 answers
RocksDB is not freeing up space after a delete
Most of our services are using a Kafka store, which as you know is using RocksDB under the hood. We are trying to delete outdated and wrongly formatted records every 6 hours, in order to free up space. Even though the record gets deleted from…
iliev951
- 11
- 2
1
vote
1 answer
To close or to not close RocksDB Cache and WriteBufferManager in kafka streams app
I am currently playing around with a custom RocksDB configuration in my streams app by extending RocksDBConfigSetter interface. I see conflicting documentation around closing cache & writeBufferManager instances.
Right now, I see that the javadoc &…
Wizard
- 1,022
- 2
- 13
- 39
1
vote
1 answer
Having consumer issues when RocksDB in flink
I have a job which consumes from RabbitMQ, I was using FS State Backend but it seems that the sizes of states became bigger and then I decide to move my states to RocksDB.
The issue is that during the first hours running the job is fine, event after…
Alejandro Deulofeu
- 332
- 1
- 11
1
vote
1 answer
How to use RocksDB tailing iterator?
I am using RocksDB Java JNI and would like to get new entries as they are added to the RocksDB.
Thread t = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
try {
System.out.println("Putting " + i);
…
JavaTechnical
- 5,807
- 8
- 40
- 79
1
vote
1 answer
Suggestion on RocksDB Configuration
I am looking for suggestions on my RocksDB configuration. Our use case is to load 100GB of key-value pairs into rocksdb and at run time only serve the key-value pairs in the db. Key is 32 bytes and value is 1.6 KB in size.
What we have right now is…
codegenx
- 13
- 1
- 7
1
vote
1 answer
RocksDB iterator seek until last matching prefix
How do I tell a RocksDB iterator to seek until the last matching prefix?
In Clojure using the RocksDB Java API:
(import '(org.rocksdb RocksDB Options ReadOptions RocksIterator Slice))
(let [opts (-> (ReadOptions.)
…
Petrus Theron
- 25,051
- 30
- 137
- 263
1
vote
1 answer
Is rocksdb WriteBatch thread safe?
For example, one class has a private db, and a public method-put .
We got several threads to run the put, is this ok ? (ps: we don't consider the order of the k/v )
void put(List values) {
WriteOptions writeOpt = new…
jcyan
- 51
- 5
1
vote
0 answers
RocksDB JNI Slow Read Performance
I am using RocksDB JNI and I found that reads are getting exponentially slow for this program
Initially, it's in an acceptable range. But when the program runs so that the total records reach 1 million then the read time logger prints are showing…
RSSAH
- 63
- 8
1
vote
2 answers
Creating RocksDB SST file in Java for bulk loading
I am new to RocksDB abd trying to create a SST file in Java for bulk loading. Eventual usecase is to create this in Apache Spark.
I am using rocksdbjni 6.3.6 in Ubuntu 18.04.03
I am keep getting this error,
org.rocksdb.RocksDBException: Keys must be…
Saba
- 11
- 2
1
vote
1 answer
Kafka stream window aggregation almost working
I need to aggregate some sensor data and I'm experimenting with windowed aggregation for a PoC/study project.
After researching here and there and a lot of attempts, I came up with the following code, which seems to be working.
I get in fact the…
Pinco Pallino
- 887
- 1
- 6
- 15
1
vote
1 answer
Does rocksdbjava api library contain the rocksdb database itself
I am very new to rockdb and will be using the rocksdb in my application as a lookup service. Does the rockDBjava library api provided contain the database itself, I mean is it not necessary to install rocksDB database separately.
I tried running the…
Tom Thorpe
- 11
- 1
1
vote
1 answer
Kafka kstream-kstream joins with sliding window memory usage grows over the time till OOM
I'm having a problem using kstream joins. What i do is from one topic i seperate 3 different types of messages to new streams.
Then do one innerjoin with two of the streams which creates another stream, finally i do a last leftjoin with the new…
kambo
- 99
- 1
- 9