Request - Not all points mapping in Berkeley Mapper

[falco-rk]

Help us understand your request (check below):

• error or bug (please include enough specific information to reproduce)
• other

Describe what you're trying to do

Not all of OGL's records are appearing on Berkeley Mapper. I searched for everything in OGL:Genomic then went to Tools->Berkeley Mapper and only 10,000 records are appearing.
Link to map.
Link to search.

@happiah-madson @dustymc @Jegelewicz @ekrimmel

[mkoo]

first, to your example Berkeleymapper can only map what is loaded in the results -- so if your search results are 10000 then it can only map 10000 georeferenced data points, which may explain that results

second, there may be some residual issues with the DNS swap (but I doubt it, still worth checking) so will alert Deck

third, I can check with Deck about elasticsearch options and add need to map all a collections mappable points.

[mkoo]

just spoke with service developer john deck: @dustymc there's a cap to how many records are sent to BM right now so can that be changed to 100,000 for now please? It was set at 10k when we had more performative concerns with BM (old conversation wtih Deck)

@dustymc where's the form I could test this out?

[dustymc]

100,000 for now please

I seriously doubt it, but I guess I'm willing to try if you're willing to accept some near-inevitable crashy-flavored downtime....

Exchanging these data as JSON (I have native tools) instead of XML (I have an expensive+dicey home-grown wrapper-upper) would radically change the outlook for me, and I'm kinda suspecting that node.js is better at handling JS than XML anyway. Is that possible?

[mkoo]

Exchanging these data as JSON (I have native tools) instead of XML (I have an expensive+dicey home-grown wrapper-upper) would radically change the outlook for me, and I'm kinda suspecting that node.js is better at handling JS than XML anyway. Is that possible?

@jdeck88 will know!

[mkoo]

@jdeck88 meeting today: @dustymc go ahead and try to post as CSV instead of Tab-delimited; should still work. In the Arctos demo of BM, it is loading over 39k points speedily so should not be a problem with 5k points

BM is now fully parsing coordinates into the GADM countries, states, counties in the Stats tool. I can demo this at the next AWG meeting.

[dustymc]

demo of BM, it is loading over 39k points

I'm not worried about melting BM, I'm worried about melting Arctos!

not be a problem with 5k points

????

ANYWAY: Arctos is now feeding BM CSV, and I've cleaned up the code to use only the cache (but with a necessary restricted locality check). I've moved the limit to 50K, but haven't been able to push that very hard because of test DB limitations. It's unfast but I haven't been able to outright kill anything. I'm not entirely convinced that this won't occasionally team up with something to overload Arctos, but it's much more efficient than it was, we now have better and more resilient hardware, and I can't break it by clicking frantically so if @mkoo is OK with this then I am too!

There are some significant stability and simplification implications involved in this, and I can't really "test" except by doing it, so I'm jumping the gate and moving this to dev for next release.

Please thoroughly test any BerkeleyMapper related functionality as soon as possible.

Let me know if you manage to break it.