DrupalCamp le 19 avril à Paris

The chat session:

 
 03:19 ustreamer-22498 : hi everybody
 03:20 ustreamer-22498 : we gonna launch the session in about 30mn....
 03:20 ustreamer-22498 : STAGING SESSION AROUND 4PM PARIS TIME
 03:40 ustreamer-80914 : so another 20 minutes or so?
 03:40 ustreamer-62557 : yes perhaps 15
 03:41 ustreamer-62557 : keep tuned
 03:57 ustreamer-80914 : bonjour salut - greatings from ontario
 03:57 ustreamer-23409 : greeting
 03:58 ustreamer-23409 : In French sorry
 03:58 ustreamer-23409 : we chat
 03:58 ustreamer-80914 : i will try to keep up - my frenchi is not what it used to be
 03:58 ustreamer-23409 : in English
 03:59 ustreamer-23409 : the slide is in English
 03:59 ustreamer-80914 : I will at least learn some new technical vocabulary
 03:59 ustreamer-80914 : en francais
 04:00 ustreamer-30191 : Ok I will try to translate for you guys
 04:00 ustreamer-23409 : ok nice
 04:00 ustreamer-30191 : Hello
 04:00 ustreamer-30191 : We are  3 persons from France24
 04:01 ustreamer-30191 : for Internet we use only Drupal
 04:01 ustreamer-23409 : since december only drupal
 04:01 ustreamer-23409 : 4 million users
 04:01 ustreamer-30191 : 4 Million unique visitors
 04:01 ustreamer-30191 : We have A CDN a Frontal Cache from Akamai
 04:01 ustreamer-23409 : more connected users
 04:01 ustreamer-30191 : we are going to go direct because of user generated content
 04:02 ustreamer-30191 : so scaling is becoming important
 04:02 ustreamer-23409 : we can never stop
 04:02 ustreamer-30191 : we are 24h network so we can not have down time
 04:02 ustreamer-30191 : this is why we need a secured staging system
 04:02 ustreamer-30191 : we do not want to take changes directly to produciton
 04:03 ustreamer-30191 : we need a way to test changes (modules added and such) with the same data as on production
 04:03 ustreamer-30191 : all without downtime
 04:03 ustreamer-67311 : The pdf file http://www.mikiane.com/files/synchroStaging.pdf
 04:04 ustreamer-30191 : We have two types of MEP >> staging to Production
 04:04 ustreamer-30191 : 80% simple changes without big effects on the database
 04:04 ustreamer-30191 : we have snapshots of the produciton of env
 04:05 ustreamer-30191 : we move them to the snapshot .... then we deploy the source code (on a NAS server)
 04:05 ustreamer-30191 : But we had the problem of staging to producion when it is complex... A new ;odule that will impact the DB directly
 04:06 ustreamer-30191 : it can ;odify existing tables, parameter some blocks, views... etc
 04:06 ustreamer-30191 : how to move to production from staging... that is testable that does not imact the produciton server
 04:07 ustreamer-30191 : There is production server that has a redundant database that is synchronized
 04:07 ustreamer-30191 : just for fail server
 04:07 ustreamer-30191 : fail over, sorry
 04:07 ustreamer-30191 : then we replicate this passive server towards the staging one
 04:08 ustreamer-30191 : each record onproduciton server is Even
 04:08 ustreamer-30191 : On staging each record has an Odd ID
 04:08 Gbec : the current pdf in live http://www.mikiane.com/files/synchroStaging.pdf
 04:08 ustreamer-30191 : so Production content is Always even
 04:08 ustreamer-30191 : so produciton info does not modify staging stuff.
 04:09 ustreamer-30191 : staging does not get replicated back to produciton auto;atically
 04:09 ustreamer-30191 : We do not change parameters on producion server
 04:09 ustreamer-30191 : So staging ... is Production + the New changes on staging
 04:10 ustreamer-30191 : then we very simply point the virtual server towards staging which becomes Production
 04:10 ustreamer-30191 : Then on the next Staging operation....
 04:10 ustreamer-30191 : we do it all over again
 04:11 ustreamer-30191 : so this is not a permanenet structure... but a ad hoc solution every time we need to move comlpex stuff to production
 04:11 ustreamer-30191 : Let's get more technical now (Guillaume)
 04:12 Gbec : http://drupal.org/node/181128
 04:12 ustreamer-32297 : http://drupal.org/node/181128#comment-279298
 04:12 Gbec : topic talking zbout technique
 04:13 ustreamer-30191 : So this is more then just Even/Odd
 04:13 ustreamer-30191 : In the Sequences table we have all the ids that are used
 04:13 ustreamer-30191 : We use Runkit
 04:13 ustreamer-30191 : that permits to redefine dynamically a function
 04:14 ustreamer-30191 : so we can "hack" the db code that does not have a hook
 04:16 ustreamer-30191 : we modify the "Next Id" by incementing in a certain way, (could be even/odd) but if you start with an existing database yyou might use a ten based system... so you incement by 10 the produciton and staging (staging would be 81 91 ... and produciton 82 92 ....
 04:16 ustreamer-30191 : )
 04:16 ustreamer-30191 : Explanation follows on the architecture you can see on the slides.
 04:16 ustreamer-30191 : We learnt that there are tables we can not replicate
 04:16 ustreamer-30191 : like cache....
 04:16 ustreamer-80914 : and session
 04:17 ustreamer-30191 : So 1. we have a replication between production and passif failover server ..
 04:17 ustreamer-30191 : 2. then we have replication between passif to staging server
 04:17 ustreamer-30191 : so staging is always synchronized with produciton
 04:18 ustreamer-30191 : so the node onproducion is ID 101 but when coming to the staging the id will be 102
 04:19 ustreamer-30191 : as when I  create content on staging I call DbNextID I can modify this value and not have conflicts
 04:19 ustreamer-30191 : but there are tables that do not use ids
 04:19 ustreamer-30191 : So how do you avoid conflicts there...
 04:20 ustreamer-30191 : Sometimes a module will do stuf like reinserting zhen they should have done insert or replace
 04:21 ustreamer-30191 : so we try to find a solution for each module that does use it... for an example for files ... (so we don't have a problem) we use the same "files" structure....
 04:21 ustreamer-30191 : usinf nfs
 04:22 ustreamer-30191 : Again... all the changes are done on staging.. then when we like the changes we just modify the virtual host to make it a live server.
 04:23 ustreamer-30191 : for those modules that make a problem... for the cache it is simple to tell the slave not to replicate...
 04:24 ustreamer-30191 : IGNORE TABLE
 04:24 ustreamer-30191 : --ignore-table
 04:25 ustreamer-30191 : so now we no longer have a staging.... until the next time we need
 04:26 ustreamer-30191 : but we do not destroy it.... so we can go back to the "old production" server
 04:27 ustreamer-30191 : It is not really easy to make all of this in a script...
 04:30 ustreamer-32297 : the replication is not total
 04:30 ustreamer-30191 : problems with possible replication of unwanted stuf...
 04:31 ustreamer-32297 : some table are not repliquade
 04:31 ustreamer-32297 : not the bloks
 04:31 ustreamer-30191 : probably we should not repliacte blocks
 04:31 ustreamer-30191 : or views...
 04:31 ustreamer-30191 : so we try to know what are the administration tables and which are the content modules...
 04:32 ustreamer-30191 : question.. how long does the staging take...
 04:32 ustreamer-30191 : ?
 04:32 ustreamer-80914 : So the architecture is Prod-Master --replication--> Prod-Slave -- replication (incomplete) i.e. minus cache and sessions --> Staging-Master -- replication (total) --> Staging-Slave
 04:33 ustreamer-32297 : in prod no admin$
 04:33 ustreamer-30191 : usually it takes 1 to 2 days
 04:34 ustreamer-30191 : 90% of simple productions.... 10% complex ones
 04:35 ustreamer-30191 : we try to package together the changes
 04:36 ustreamer-30191 : again you can see:http://drupal.org/node/181128
 04:36 ustreamer-32297 : we will try to post a update
 04:36 ustreamer-30191 : we will try to get more information... about the specific problems we met
 04:36 ustreamer-30191 : like with the locale module
 04:36 ustreamer-30191 : where sometimes we got 500 mb of data for replication when the cache was regenerated..
 04:37 ustreamer-32297 : Question / answer ?
 04:37 ustreamer-30191 : If you have questions .. you are invited...
 04:37 ustreamer-80914 : Do you have a scripted solution to initiate the staging replica?
 04:38 ustreamer-30191 : On a simple moving to produciton ... solutions for updating css...
 04:39 ustreamer-30191 : add a cache buster
 04:39 ustreamer-30191 : with the version id= version id
 04:41 ustreamer-30191 : For the scripted solutions:
 04:41 ustreamer-30191 : Yes we have but it is still not excellent...
 04:41 ustreamer-32297 : A beging a solution of scripting
 04:41 ustreamer-32297 : if you want to know more
 04:41 ustreamer-32297 : you can get in touch to works on packaging
 04:41 ustreamer-30191 : We want to try to help package
 04:41 Gbec : @80914 we got a beging of solution but not really stable
 04:41 ustreamer-32297 : at the end of slide
 04:41 ustreamer-32297 : you have contact
 04:42 ustreamer-30191 : in the presentaion you can find our emails
 04:42 ustreamer-30191 : Another question:
 04:42 ustreamer-30191 : This is bringing production to preproduction
 04:43 ustreamer-32297 : mlevypro@gmail.com as a contact
 04:44 ustreamer-32297 : other solution
 04:46 ustreamer-32297 : are explained
 04:47 ustreamer-30191 : rue89 also have a nice solution that can be mixed with ours...
 04:47 ustreamer-30191 : I do not have details yet..
 04:47 ustreamer-30191 : they use macro of Devel module
 04:48 ustreamer-30191 : some stuff does not get registered by Macro...
 04:48 ustreamer-30191 : then they have a system that replays this in an update using the hook
 04:48 ustreamer-32297 : an other solution update with macro
 04:48 ustreamer-30191 : drupal_macro_execute if I remember right...
 04:49 ustreamer-32297 : other question ?
 04:50 ustreamer-80914 : I heard the mention of views a few times - The best way to deal with views is to define them exclusively in code - and NOT use the administration UI.
 04:51 ustreamer-30191 : Put configuration in TEXT files
 04:52 ustreamer-30191 : we know how to do production on text based files...
 04:52 ustreamer-30191 : we have work to do...
 04:52 ustreamer-30191 : can we use this solutions with load balancing systems?
 04:52 ustreamer-30191 : difficut to answer offhand
 04:53 ustreamer-30191 : Session Ending....
 04:53 ustreamer-30191 : Thank you all
 04:53 ustreamer-32297 : thank you
 04:53 Gbec : thx everyody
 04:53 ustreamer-80914 : I meant defining views is hook_views_default_views() - but exports in text files is doable too.
 04:53 ustreamer-32297 : conclusion
 04:53 ustreamer-80914 : thank you