From monoliths to clusters: transforming ALA s infrastructure

Transcription

1 From monoliths to clusters: transforming ALA s infrastructure Matt Andrews DevOps Manager 18 October 2018 The ALA is made possible by contributions from its many partners. It receives support through the Australian Government through the National Collaborative Research Infrastructure Strategy (NCRIS) and is hosted by CSIRO.

2 From monoliths to clusters Matt Andrews 2

3 From monoliths to clusters Matt Andrews 3

4 From monoliths to clusters Matt Andrews 4

5 From monoliths to clusters Matt Andrews 5

6 From monoliths to clusters Matt Andrews 6

7 From monoliths to clusters Matt Andrews 7

8 From monoliths to clusters Matt Andrews 8

9 From monoliths to clusters Matt Andrews 9

10 Atlas core services BIE species search, taxonomy Biocache occurrences Collectory natural history collections DOI data downloads for publication Spatial portal spatial analysis Species lists User authentication From monoliths to clusters Matt Andrews 10

11 From monoliths to clusters Matt Andrews 11

12 Atlas usage Over the last year: 1.1 million user sessions 345,000 unique visitors 85% from Australia From monoliths to clusters Matt Andrews 12

13 From monoliths to clusters Matt Andrews 13

14 From monoliths to clusters Matt Andrews 14

15 Occurrence records An event: observation of a species Over 78 million in the ALA Typically ~1000 fields per record Over 120,000 species From monoliths to clusters Matt Andrews 15

16 More ALA stats Over 400 spatial layers Over 1 million images Over 1.5 million downloads Over 6000 datasets Over 40,000 users From monoliths to clusters Matt Andrews 16

17 Atlas core services BIE species search, taxonomy Biocache occurrences Collectory natural history collections DOI data downloads for publication Spatial portal spatial analysis Species lists User authentication From monoliths to clusters Matt Andrews 17

18 From monoliths to clusters Matt Andrews 18

19 From monoliths to clusters Matt Andrews 19

20 From monoliths to clusters Matt Andrews 20

21 From monoliths to clusters Matt Andrews 21

22 From monoliths to clusters Matt Andrews 22

23 From monoliths to clusters Matt Andrews 23

24 From monoliths to clusters Matt Andrews 24

25 From monoliths to clusters Matt Andrews 25

26 From monoliths to clusters Matt Andrews 26

27 From monoliths to clusters Matt Andrews 27

28 From monoliths to clusters Matt Andrews 28

29 From monoliths to clusters Matt Andrews 29

30 From monoliths to clusters Matt Andrews 30

31 From monoliths to clusters Matt Andrews 31

32 From monoliths to clusters Matt Andrews 32

33 Biocache data store: Apache Cassandra search: Apache Solr biocache applications From monoliths to clusters Matt Andrews 33

34 Biocache data store: Apache Cassandra search: Apache Solr biocache applications: biocache-hub: front end (browsers) biocache-service: web services biocache-store: data handling From monoliths to clusters Matt Andrews 34

35 Old biocache one Cassandra server one Solr server one biocache-hub server a pair of servers for biocache-service biocache-store with Cassandra From monoliths to clusters Matt Andrews 35

36 Data cycle data quality checks initial import into Cassandra processing environmental sampling fully processed records in Cassandra generate indexes for Solr From monoliths to clusters Matt Andrews 36

37 Old biocache any maintenance caused downtime any server issue caused downtime reaching hard limits on data size data cycle extremely slow and brittle From monoliths to clusters Matt Andrews 37

38 Old biocache any maintenance caused downtime any server issue caused downtime reaching hard limits on data size data cycle extremely slow and brittle: processing and sampling: six days indexing: 24 hours From monoliths to clusters Matt Andrews 38

39 Old biocache any maintenance caused downtime any server issue caused downtime reaching hard limits on data size data cycle extremely slow and brittle: processing and sampling: 6 days indexing: 24 hours no scalability From monoliths to clusters Matt Andrews 39

40 Designing a solution From monoliths to clusters Matt Andrews 40

41 Designing a solution latest versions of Cassandra and Solr Cassandra cluster Solr Cloud cluster applications in load balanced pools From monoliths to clusters Matt Andrews 41

42 IO improvements Cassandra from 1 to 4 nodes From monoliths to clusters Matt Andrews 42

43 IO improvements Cassandra from 1 to 4 nodes biocache-store to 4 separate servers From monoliths to clusters Matt Andrews 43

44 Building the solution Cassandra: single-node version 1 to clustered version 3 Solr: single-node version 4 to clustered version 6 From monoliths to clusters Matt Andrews 44

45 From monoliths to clusters Matt Andrews 45

46 Initial UK version Cassandra: four nodes Solr: eight nodes Fast, fault-tolerant data ingestion Larger number of occurrences, but much smaller width (number of fields) From monoliths to clusters Matt Andrews 46

47 AU version development Around 18 months elapsed, 2-5 developers Fewer occurrence records, but much wider data (more fields) Needed to maintain backwards compatibility for most types of request From monoliths to clusters Matt Andrews 47

48 AU version development Around 18 months elapsed, 2-5 developers Fewer occurrence records, but much wider data (more fields) Needed to maintain backwards compatibility for most types of request Iterative configuration testing in multiple components From monoliths to clusters Matt Andrews 48

49 AU version development Performance initially much lower than legacy Eventually got performance to similar level Costs slightly higher Switched to new version in July 2018 From monoliths to clusters Matt Andrews 49

50 Data cycle changes Processing and sampling: Old system - 6 days, often repeated due to low fault tolerance From monoliths to clusters Matt Andrews 50

51 Data cycle changes Processing and sampling: Old system - 6 days, often repeated due to low fault tolerance New system 11 hours, highly fault tolerant From monoliths to clusters Matt Andrews 51

52 Data cycle changes Processing and sampling: Old system - 6 days, often repeated due to low fault tolerance New system 11 hours, highly fault tolerant Indexing: Old system 24 hours From monoliths to clusters Matt Andrews 52

53 Data cycle changes Processing and sampling: Old system - 6 days, often repeated due to low fault tolerance New system 11 hours, highly fault tolerant Indexing: Old system 24 hours New system 3.5 hours Now indexing every few days From monoliths to clusters Matt Andrews 53

54 Robustness clustered Cassandra and Solr can now tolerate having a node go down biocache applications (biocache-hub and biocache-service) now pooled behind load balancers, can tolerate a server going down From monoliths to clusters Matt Andrews 54

55 Scalability expanding Cassandra or Solr capacity is relatively straightforward expecting to be able to handle future major expansion of data width (species traits) From monoliths to clusters Matt Andrews 55

56 Biocache data store: Apache Cassandra search: Apache Solr biocache applications From monoliths to clusters Matt Andrews 56

57 Thank you Matt Andrews DevOps Manager Atlas of Living Australia The ALA is made possible by contributions from its many partners. It receives support through the Australian Government through the National Collaborative Research Infrastructure Strategy (NCRIS) and is hosted by CSIRO.