Nosql Solution To Store 20[tb] Of Data, As Vector/array?

I need to build a system to efficiently store & maintain a huge amount (20 [TB]) of data (and be able to access it in 'vector' form). Here are my dimensions: (1) time (given a

Solution 1:

MongoDB scales awesomely and supports many of the indexing features you'd typically find in an RDBMS such as compound key indexes. You can use a compound index on the name and time attributes in your data. Then you can retrieve all instrument readings with a particular name and date range.

[Now in the simple case where you're strictly interested in just that one basic query and nothing else, you can just combine the name and timestamp and call that your key, which would work in any key-value store...]

HBase is another excellent option. You can use a composite row key on name and date.

As others have mentioned, you can definitely use a relational database. MySQL & PostgreSQL can certainly handle the load and table partitioning might be desirable in this scenario as well since your dealing with time ranges. You can use bulk loading (and disabling indexes during loading) to decrease insertion time.