Enhancement
Speed up the Topincs API
Issue description
On large data sets the Topincs API does not perform well enough.
Developer comments
After a week of on and off research, it became clear that apc only offers a speed advantage over the filesystem when the stored variables are small.
A test array of 6.2MB (in serialized format) was as slow to maintain (read/write) from APC as in the filesystem (including serialization and deserialization). Even disk speed did not seem to matter much. Tried SSD, HD and RAMDISK.
Currently i am storing a full index for a statement type. If there is 1 mio occurrences of that type, the array has 1 mio entries. Given the above result, i have to rethink that.
Implemented two memcached implementations for storing the index, one which passed changes directly to memcached and reads always from memcached. Another where memcached works just as a final storage for reading and writing (the complete index). It is based on the current file based index implementation. In the prior version building the index was slowed down significantly, since i always have to read from memcached before i can but the statement in the index. I cannot see how to do it differently in memcached, which makes it a show stopper. Luckily Redis has a Hash datatype which could work well.
That said, the latest improvement ([6409, see this change]) makes the current file based indexing system better. The only minus is that a large index takes long to get ready (by reading the file and unserzializing it) and memory consumption is. Check out var_export.
On large indices it still is useful to try to shift them to an in-memory database, since just reading 10 large indices might take 1 sec. Actually that is less of a problem as it might be more important to speed up the actual retrival of the data. In long running computations this 1 second does not matter much.
For a considerable approach with pure PHP means (as of 7) and without serialization see [https://blog.graphiq.com/500x-faster-caching-than-redis-memcache-apc-in-php-hhvm-dcd26e8447ad, this article].
The opcache method does work quite well and considerably reduces the time necessary to retrieve an index (once the index file has been hit). Currently indices are maintained per statement type. It might been worth considering to maintain additionally (or instead) an index per tobject. Reads in the API then could be satisfied by one opcached file fetch. It is necessary though to check if long running services (e.g. searches) do indeed benefit from this. Or what is it exactly that causes these services to be sluggish. Mostly it occurs when many tobjects are iterated over.
|
Work sessions24
Start |
2015-03-03T09:47:16
|
End |
2015-03-03T14:38:42
|
Participant |
Robert Cerny
|
Start |
2015-03-04T08:29:03
|
End |
2015-03-04T10:21:04
|
Participant |
Robert Cerny
|
Start |
2015-03-05T09:51:03
|
End |
2015-03-05T12:23:16
|
Participant |
Robert Cerny
|
Start |
2015-03-05T14:23:28
|
End |
2015-03-05T16:50:23
|
Participant |
Robert Cerny
|
Start |
2015-03-05T22:20:34
|
End |
2015-03-05T22:45:00
|
Participant |
Robert Cerny
|
Start |
2015-03-07T16:31:27
|
End |
2015-03-07T17:54:51
|
Participant |
Robert Cerny
|
Start |
2015-03-08T07:44:30
|
End |
2015-03-08T08:56:31
|
Participant |
Robert Cerny
|
Start |
2015-03-08T13:57:17
|
End |
2015-03-08T20:20:49
|
Participant |
Robert Cerny
|
Start |
2015-03-09T10:57:33
|
End |
2015-03-09T16:59:59
|
Participant |
Robert Cerny
|
Start |
2015-03-10T05:00:26
|
End |
2015-03-10T07:26:55
|
Participant |
Robert Cerny
|
Start |
2015-03-11T14:54:18
|
End |
2015-03-11T16:54:18
|
Participant |
Robert Cerny
|
Start |
2015-03-12T10:09:06
|
End |
2015-03-12T13:41:26
|
Participant |
Robert Cerny
|
Start |
2015-03-12T14:13:41
|
End |
2015-03-12T15:28:22
|
Participant |
Robert Cerny
|
Start |
2015-03-13T10:24:51
|
End |
2015-03-13T12:25:02
|
Participant |
Robert Cerny
|
Start |
2015-03-14T07:06:11
|
End |
2015-03-14T10:16:11
|
Participant |
Robert Cerny
|
Start |
2015-03-16T14:26:12
|
End |
2015-03-16T15:47:57
|
Participant |
Robert Cerny
|
Start |
2015-04-04T08:24:10
|
End |
2015-04-04T09:59:08
|
Participant |
Robert Cerny
|
Start |
2015-04-05T10:48:26
|
End |
2015-04-05T13:03:52
|
Participant |
Robert Cerny
|
Start |
2015-04-05T13:13:31
|
End |
2015-04-05T14:52:12
|
Participant |
Robert Cerny
|
Start |
2015-04-05T15:15:18
|
End |
2015-04-05T18:11:56
|
Participant |
Robert Cerny
|
Start |
2015-04-06T08:01:30
|
End |
2015-04-06T10:16:19
|
Participant |
Robert Cerny
|
Start |
2015-04-12T11:40:16
|
End |
2015-04-12T13:20:23
|
Participant |
Robert Cerny
|
Start |
2019-08-15T21:02:04
|
End |
2019-08-15T22:02:16
|
Participant |
Robert Cerny
|
Start |
2019-08-16T08:22:30
|
End |
2019-08-16T13:53:45
|
Participant |
Robert Cerny
|
|
We are sorry
This page cannot be displayed in your browser. Use Firefox, Opera, Safari, or Chrome instead.