Migrating from App Engine pull duties to Cloud Pub/Sub (Module 19)

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

The Serverless Migration Station collection is aimed toward serving to builders modernize their apps operating one among Google Cloud’s serverless platforms. The previous (Migration Module 18) video demonstrates the best way to add use of App Engine’s Activity Queue pull duties service to a Python 2 App Engine pattern app. At the moment’s Module 19 video picks up from the place that leaves off, migrating that pull job utilization to Cloud Pub/Sub.

Transferring away from proprietary App Engine providers like Activity Queue makes apps extra transportable, giving them sufficient flexibility to:

 

Understanding the migrations

Module 19 consists of implementing three totally different migrations on the Module 18 pattern app:

• Migrate from App Engine NDB to Cloud NDB
• Migrate from App Engine Activity Queue pull duties to Cloud Pub/Sub
• Migrate from Python 2 to Python (2 and) 3

The NDB to Cloud NDB migration is an identical to the Module 2 migration content material, so it is not coated in-depth in Module 19. The unique app was designed to be Python 2 and three suitable, so there is no work there both. Module 19 boils down to a few key updates:

• Setup: Allow APIs and create Pub/Sub Matter & Subscription
• How work is created: Publish Pub/Sub messages as an alternative of including pull duties
• How work is processed: Pull messages as an alternative of leasing duties

Other than these bodily modifications, a key hurdle to beat is knowing the variations in terminology between pull duties and Pub/Sub. The next chart makes an attempt to demystify this so builders can extra simply grasp how they differ:

Terminology variations between App Engine pull duties and Cloud Pub/Sub

Reflecting the chart, these variations may be summarized like this:

1. With Pull Queues, work is created in pull queues whereas work is distributed to Pub/Sub matters
2. Activity Queue pull duties are referred to as messages in Pub/Sub
3. With Activity Queues, employees entry pull duties; with Pub/Sub, subscribers obtain messages
4. Leasing a pull job is identical as pulling a message from a Pub/Sub subject by way of a subscription
5. Deleting a job from a pull queue while you’re accomplished is analogous to efficiently acknowledging a Pub/Sub message

The video walks builders via the terminology in addition to the code modifications described above. Under is pseudocode implementing the important thing modifications to the principle utility (new or up to date traces of code bolded):

Migration from App Engine Activity Queue pull duties to Cloud Pub/Sub

Observe how many of the code, particularly app operations and information fashions are left comparatively unchanged. The one seen modifications are switching from App Engine NDB and Activity Queue to Cloud NDB and Pub/Sub. Full variations of the app earlier than and after making the modifications may be discovered within the Module 18 and Module 19 repo folders, respectively. Along with the video, make sure you take a look at the Module 19 codelab which leads you step-by-step via the migrations mentioned.

Wrap-up

Module 19 encompasses a migration of App Engine pull duties to Cloud Pub/Sub, however builders ought to word that Pub/Sub itself just isn’t primarily based on pull duties. It’s a fully-featured asynchronous, scalable messaging service that has many extra options than the pull performance offered by Activity Queue. For instance, Pub/Sub has different options like streaming to BigQuery and push performance. Pub/Sub push operates otherwise than Activity Queue push duties, therefore why we suggest push duties be migrated to Cloud Duties as an alternative (see Module 8). For extra info on all of its options, see the Pub/Sub documentation. As a result of Cloud Duties does not assist pull performance, we flip to Pub/Sub as an alternative for pull job customers.

Whereas we suggest customers transfer to the most recent choices from Google Cloud, neither of these migrations are required, and do you have to decide to take action, can do them by yourself timeline. In Fall 2021, the App Engine crew prolonged assist of most of the bundled providers to 2nd technology runtimes (which have a 1st technology runtime), which means you do not have emigrate to standalone Cloud providers earlier than porting your app to Python 3. You may proceed utilizing Activity Queue in Python 3 as long as you retrofit your code to entry bundled providers from next-generation runtimes.

For those who’re utilizing different App Engine legacy providers make sure you take a look at the opposite Migration Modules on this collection. All Serverless Migration Station content material (codelabs, movies, supply code [when available]) may be accessed at its open supply repo. Whereas our content material initially focuses on Python customers, the Cloud crew is engaged on protecting different language runtimes, so keep tuned. For extra video content material, take a look at our broader Serverless Expeditions collection.