Bhanu Singh - Web developer working mainly on WordPress.

🕷️ ScrapeWave: Web Scraping using Puppeteer in a Docker container with a UI dashboard and database.

Bhanu Singh — Sat, 11 May 2024 08:01:50 GMT

https://github.com/git-bhanu/scrapewave

🕷️ ScrapeWave project was created to help me accomplish some scraping tasks.

The requirements of the task I was building were:

To be able to access a particular link.
Parse the content of the page and save it somewhere.
Log error in case something unexpected happens.
Ability to start and stop the scarping.
Ability to add more data into the tool.
Ability to import and export data into and from the tool.
And, provide a UI for the person who is administering the tool to be able to see the progress.
Easy & quick to deploy multiple instances of it.

This project uses multiple technologies to achieve the requirement.

Server
1. ViteExpress - @vitejs integration module for @expressjs.
2. SQlite - SQLite client wrapper around sqlite3 for Node.js applications with SQL-based migrations API written in Typescript.
3. Puppeteer - Node.js API for Chrome
4. https://github.com/socketio/socket.io - Realtime application framework (Node.JS server).
5. Other packages worth mentioning are:-
  - axios
  - fast-csv
UI
1. VueJs - Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web
2. Pinia - Intuitive, type safe, light and flexible Store for Vue using the composition api with DevTools support.
3. Vuetify - Vue Component Framework
Docker - Accelerated Container Application Development

ScrapeWave in action

Adding data to your tool

Using CSV import you can add data to your scarper.

Head over to upload the page and upload the data.

You can use the sample.csv file attached to the repository.

https://github.com/git-bhanu/scrapewave/blob/main/sample.csv

As you can see we have three data which can be seen in the table at home.

Starting the script

You can start the scraping process by clicking on the start button on the home page.

As you can see from the image above scraping for the first two entries is done. We also nice logs in your terminal which shows the current status of the software.

Deploying ScrapeWave in Cloud

Once you have created the changes as per your liking in the project (learn how to do that from the readme file of the repo) you can create a docker image.

You can name the image whatever you want.

docker build -t bhanu/scrapewave:latest .

Here, bhanu would be the docker hub username and scrapewave is the name of the image.

You can push the image to your docker hub using the following command.

Make sure you are logged in your dokcer hub account.

dokcer push bhanu/scrapewave:latest

Once it is done, go to the server using ssh and install two software, docker and Nginx.

To install Docker follow the steps here:

https://www.digitalocean.com/community/questions/how-to-install-and-run-docker-on-digitalocean-dorplet

To install Nginx follow the steps here:

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04

Once you have successfully installed both,

You can use docker run -dit --name -p :3000 krenovate/data-validator:latest to create a container.

For eg.

docker run -dit -v instance-1:/usr/src/app/db --name instance-1 -p 8080:3000 --restart on-failure bhanu/scrapewave:latest

This command does quite a few things:

Creates a container using latest tag of bhanu/scrapewave
Mounts the SQlite database in your host filesystem, ensuring even if you recreate the the container with instance-1 volume your data would still be persistent.
For any scenario if the container exists, it will try to restart the container so that your scraping doesn't stop.

I plan to keep on improving this project so that it helps me and other folks to use it as the base to get up and running with there scraping task.

Feel free to raise issues if you see anything which is broken - https://github.com/git-bhanu/scrapewave/issues

Also any kind of contribution is appreciated.

How to Upgrade to a non-LTS Ubuntu [23.10] ?

Bhanu Singh — Thu, 11 Apr 2024 14:33:58 GMT

You can do the following to upgrade your Ubuntu version to 23.04 version.

First, we have to change the Prompt from lts to normal.

Run > sudo nano /etc/update-manager/release-upgrades and find Prompt and switch it to normal from lts.

Now to update use do-release-upgrade -c

Follow the steps as prompted and you should be good to update your Ubuntu version to version 23.10

💡

Do remember this is not an LTS (long-term support) version.

Must Read's

Bhanu Singh — Thu, 10 Nov 2022 17:26:25 GMT

Hello everyone, When I am not working at my day job, I usually surf internet and stumble upon some really good articles about tech, productivity, life and other varied topics.

Writing Down What I Do — In Obsidian [by Chris Krycho] [Productivity]
Taming Names in Software Development [by Joseph Glass] [Development]
When "Zoë" !== "Zoë". Or why you need to normalize Unicode strings [by Alessandro Segala] [Development]

Last Updated: 24 Dec, 2022

ACF : Reset all WYSIWYG inside a field group using JS

Bhanu Singh — Wed, 23 Jun 2021 18:56:58 GMT

You can find the JS code below if you are just looking for them. If you are interested in the explanation continue reading.

function bks_add_custom_js()
{
    ?>
    <script type="text/javascript">
        (function($) {
            acf.add_action('ready', function( $el ){

                function resetWYSIWYG(fieldGroup) {
                    const parent = $(fieldGroup);

                    const wysiwygs = parent.find('[data-type="wysiwyg"]');

                    $(wysiwygs).each(function(index, wysiwyg) {
                        let tinyID = $(this).find("textarea").attr("id");
                        let tinyInstance = tinyMCE.editors[tinyID];
                        if(tinyInstance) {
                            tinyInstance.setContent('');
                        }
                    });
                }

            // Reset all WYSIWYG for Field group haivng 'group_60d191ab7f442' id
            resetWYSIWYG('#acf-group_60d191ab7f442');
            });
        })(jQuery);
    script>
    'acf/input/admin_footer', 'bks_add_custom_js');

We have used acf/input/admin_footer action from acf to add the custom code on the footer of our page, wherever the acf fields are shown.

Inside the function we have written our code inlined. Once the dom is ready we use action ready from ACF JS Api .

After that we define our resetWYSIWYG function. I have commented what each line does.

function resetWYSIWYG(fieldGroup) {
     // Get main field group.
    const parent = $(fieldGroup);

    // Find all the items where data-type attribute has value of wysiwyg
    const wysiwygs = parent.find('[data-type="wysiwyg"]');

    // Loop around the found wysiwygs
    $(wysiwygs).each(function(index, wysiwyg) {
        // Find ID of the wysiwyg
        let tinyID = $(this).find("textarea").attr("id");
        // Get instance of tinyMCE
        let tinyInstance = tinyMCE.editors[tinyID]; 
        // If you get instance then use setContent to set the vaule empty.
        if(tinyInstance) {
            tinyInstance.setContent('');
        }
    });
}

Finally you can call the function with your group field id like so. As we are targeting id ACF adds acf- before group_key, so make sure you add that when you are specifying selector for the group.

resetWYSIWYG("#acf-group_60d191ab7f442");

I hope this snippet helped you. 😊

Let me know if you have any question below in the comment and I will try my best to answer them.

Creating a custom webhook in WordPress to get data from third part services

Bhanu Singh — Sun, 13 Jun 2021 18:24:47 GMT

Before setting up a webhook lets understand what a webhook is.

A webhook (also called a web callback or HTTP push API) is a way for an app to provide other applications with real-time information.

In simpler terms, they are a great way for applications to speak to each other.

For example : You have a form which you have created on typeform, and you would like to do something on your website when the form is submitted. You can use webhook to achieve this. Typeform supports webhook.

You have to add a webhook URL in your typeform form. When someone fills up the form typeform will send the data to the URL which you have entered. To recieve that data you have to setup a webhook on your wordpress website and get that data.

Setting up Webhook

We will use do_action( "admin_post_nopriv_{$action}" ) provided by wordpress . This action "fires on a non-authenticated admin post request for the given action."

add_action('admin_post_nopriv_get_typeform_data',  'process_data_form_tyepform_webhook', 10);

Here, the action name is get_typeform_data. Which means your webhook url will be

https://yourwebsite.com/wp-admin/admin-post.php?action=get_typeform_data

Processing Incoming data

Now we will define a function which would accept the data and process it.

function process_data_form_tyepform_webhook() {

   $request = file_get_contents('php://input'); // get data from webhoook

   $data = json_decode($request, true); // decode the data is you are getting data in JSON format
   // log it in debug.log
  error_log( $request );
}

Here we have used file_get_contents to store the value in the variable. Most of the time the data you will get is in the form of json, so we have decoded the data and logged it. ( Make sure your debugging is set to true. )

Now you can use the $data variable and process in any way you like. Create a post, create a user etc.

Caveat

When testing your funtionality make sure that you are not using your local development environment. You must have hosted your website somewhere it can be visited by other server. If you are interested in testing it locally you should look into https://ngrok.com/.

I hope you enjoyed reading this blog, as much as I enjoyed writing it. 😊

Let me know if you have any question below in the comment and I will try my best to answer them.

Efficient debugging workflow for WordPress developers

Bhanu Singh — Sat, 12 Jun 2021 21:33:58 GMT

When I started developing on wordpress with php I would always reach for var_dump() which would show me information about the variable. It did what it promised, it dumped the information of the variable.

For example.

$variable = array( 1, 2, 3 );
var_dump( $variable );
die(); // Stop the code. I will come back to this later.

This is rendered like this on the frontend.

This looked good enough for a simple array, if it was a multidimensional array like below

$cars = array (
  array("Volvo",22,18),
  array("BMW",15,13),
  array("Saab",5,2),
  array("Land Rover",17,15)
); // This array is taken from the popular website w3schools 😜
var_dump( $cars );
die(); // Stop the code. I will come back to this later.

Okay let's look at the output.

I am very sure that this is very difficult for most developers to scan and make sense out of 😑.

In my pursuit to make my life easier I found other solution to my problem, "Pretty printing arrays".

$cars = array (
  array("Volvo",22,18),
  array("BMW",15,13),
  array("Saab",5,2),
  array("Land Rover",17,15)
); 
print("".print_r($cars,true)."
");
die(); // Stop the code. I will come back to this later.

Output :

This is already much better. I can scan the array in a format, its easier to understand.

But my happiness didn't last for too long, when I started working with classes and objects.

I will use global variable of woocommerce to depict the issue which I faced.

global $woocommerce;
print("".print_r($woocommerce,true)."
");
die();

Output :

Okay that is readable but its very long. To give you a prespective.

This is in the lowest zoom of 25% and there is still a lot of scrolling left.

That's a lot about the problem, lets get into the solution. 😁

Solution - Debug Toolkit

The solution is a plugin called Debug Toolkit by author hellofromTonya .

This plugin is available on wordpress plugin directory. You can download it here.

How is it a solution?

It comes with three very awesome tools.

Whoops - whoops is a nice little library that helps you develop and maintain your projects better, by helping you deal with errors and exceptions in a less painful way.
Kint - Zero-setup replacement for var_dump(), print_r(), and debug_backtrace().
VarDumper - The VarDumper component provides mechanisms for extracting the state out of any PHP variables. Built on top, it provides a better dump() function that you can use instead of var_dump.

How to use it ?

Once you have installed the plugin activate it and you are ready. Nothing else is needed. You can start using it.

It is advised to use this plugin only on development and staging servers.

Let's use this plugin to dump our $wocommerce variable.

This plugin provides us with two type of dumper

vd() - from VariableDumper
d() - from Kint

Lets test both of them.

Testing `vd()`

So much better, we get color coding and toggles, we can use toggle open only those which we want. Let's try to find what our WC_Cart looks like it.

I can keep toggling on and off to read the output.

Testing `d()`

This may looks similar to vd(); but its not. You can toggle parts of output in here as well.

But it also has an Available methods tab which shows all the methods that are applicable on that particular variable.

And not only for the variable but you can also see Available methods for children values. For example we can see available methods for our cart.

This becomes so handy that I have discovered so many methods using this that I couldn't find on stackoverflow and wordpress support forum. This is like a crisp doc with variable information. And if code has a docbloc you can see that information here.

Example

It doesn't end at vd and d. You can use

vd() is an alias for vdump()
vdd() and vdd() are aliases for vdump_and_die()
d() is an alias for dump()
dd() and ddd() are aliases for dump_and_die()

What number 2 and 4 does is that it will halt the code at the dumping line and won't run it further.

Tracing

Generates a backtrace for a code is called tracing.

tracevd(); – Combines trace() and vd()
traced(); – Combines trace() and d()
tracevdd(); – Combines trace() and vdd()
tracedd(); – Combines trace() and dd()
tracevddd(); – Combines trace() and vddd()
traceddd(); – Combines trace() and ddd()

Output of tracevdd();

Apart from dumping the the variable information it also gives you the traceback.

What power it gives you, you ask ?

You can see the line of code it hit before at every trace.

You can see what was the value of the arguments at that time.

And, also Callee object whenever possible.

Errors and exception.

Whoops extenstion of this plugin makes your life a lot better by giving you a clean look when you hit an error.

On the top left hand side it gives you the error.

You get a view of code on which the error occured.

Befow that are stack frames which you can click on and see the trace of code.

You can test Whoops live here. http://filp.github.io/whoops/demo/

My opinion on XDebug

Xdebug is said to be the defacto for php development and help with a lot of things.

Step Debugging
Better var_dump()
Tracing
Profiling
Code Coverage Analysis

I think its a great tool and helps with improving developers development experience. But it can be very tough to setup for a lot of developers. I myself have tried to install and use it and haven't been very successful in doing so.

I also think that Debug Toolkit is a plugin which gives a lot of tools to developers in 0 configuration. The reason I wanted to write this blog is because of the low number of download of this plugin.

I hope you will start using it and make your life a lot easier. 😍

Please share your experience of using Debug toolkit in the comments below. You can also ask questions you may have. I will try my best to answer them.

WooCommerce : Update order automatically based on meta key

Bhanu Singh — Sat, 12 Jun 2021 12:55:27 GMT

Its a common request of woocommerce store owners to be able to change the state of order automatically when the order meta gets updated.

We will see how can be use WP Cron and wc_get_query to update order status autmatically.

1. Setting up the CRON Job.

/*
 * Trigger the code when everything is loaded.
 * */
add_action( 'init', 'bks_setup_cron_task' );

/*
 * Set a cron job for daily Midnight.
 * */
function bks_setup_cron_task() {
    if ( ! wp_next_scheduled( 'bks_update_order_status' ) ) {
        wp_schedule_event( time() , 'daily', 'bks_update_order_status' );
    }
}

// Plug the cron job with function 'bks_mark_order_complete_if_delivered'
add_action( 'bks_update_order_status', 'bks_mark_order_complete_if_delivered' );

There are multiple things going on here. Let's take a look at them one by one.

We are using add_action function with init hook which fires after WordPress has finished loading but before any headers are sent. We have passed a funtion in bks_setup_cron_task which would run when this hook is executed.

if ( ! wp_next_scheduled( 'bks_update_order_status' ) ) { ... }

Inside this function we are checking if a cron job named bks_update_order_status is scheduled or not using wp_next_scheduled function.

wp_schedule_event( time() , 'daily', 'bks_update_order_status' );

And then we use wp_schedule_event function to set when it should run next.

The functions accepts 5 parameter but we are using only the 3 which are required ones.

$timestamp (int) - Unix timestamp (UTC) for when to next run the event.
$recurrence (string) - This determines how often the event should subsequently recur. We have set it to daily. Default values you can set here are ‘hourly’, ‘twicedaily’, ‘daily’, and ‘weekly’.
$hook (string) Action hook to execute when the event is run. We have set it to a function called bks_mark_order_complete_if_delivered.

add_action( 'bks_update_order_status', 'bks_mark_order_complete_if_delivered' );

And finally using the cron hook bks_update_order_status to execute a function

You can see in the screenshot below that we have successfully registered a cron job which would execute bks_mark_order_complete_if_delivered() function.

I have used WP Crontrol plugin to monitor my cron job.

2. Writing the logic to actually update the order status

function bks_mark_order_complete_if_delivered(){
    $args = array(
        'status' => array( 'wc-processing'),
        'limit'  => -1,
        'date_created' => '>' . ( time() - 864000 ),
        'delivery_status' => 'delivered'
    );


    $orders = wc_get_orders( $args );

    foreach ($orders as $order){
        $order->update_status( 'completed' );
        $order->save();
    }
};

Here we are using wc_get_orders function where we pass 4 parameters.

status : array( 'wc-processing') - we only want order which has the status of wc-processing
limit : -1 - No limit, we want all the orders without any limit
date_created : '>' . ( time() - 864000 ) - orders whose date is greater than time() [ current time ] - 864000 seconds ( = 10 days ). ie. in last 10 days.
delivery_satus ; delivered : This is a custom paramater created which checks if the value of deliver_status order meta is delivery or not and only those orders are returned which satisfy the condition.

You can add Custom parameter like this.

function bks_handle_custom_query_var( $query, $query_vars ) {
    if ( ! empty( $query_vars['delivery_status'] ) ) {
        $query['meta_query'][] = array(
            'key' => 'delivery_status',
            'value' => esc_attr( $query_vars['delivery_status'] ),
        );
    }

    return $query;
}

add_filter( 'woocommerce_order_data_store_cpt_get_orders_query', 'bks_handle_custom_query_var', 10, 2 );

Once the args is done we get

all the orders
with processing status
in last 10 days
having delivery_status meta as delivered.

We the loop through the order, update the status and save.

$order->update_status( 'completed' );
$order->save();

Summary

So you have to finally add this combined code in your functions.php of your child theme.

/*
 * Trigger the code when everything is loaded.
 * */
add_action( 'init', 'bks_setup_cron_task' );

/*
 * Set a cron job for daily Midnight.
 * */
function bks_setup_cron_task() {
    if ( ! wp_next_scheduled( 'bks_update_order_status' ) ) {
        wp_schedule_event( time() , 'daily', 'bks_update_order_status' );
    }
}

// Plug the cron job with function 'bks_mark_order_complete_if_delivered'
add_action( 'bks_update_order_status', 'bks_mark_order_complete_if_delivered' );

function bks_mark_order_complete_if_delivered(){
    $args = array(
        'status' => array( 'wc-processing'),
        'limit'  => -1,
        'date_created' => '>' . ( time() - 864000 ),
        'delivery_status' => 'delivered'
    );


    $orders = wc_get_orders( $args );

    foreach ($orders as $order){
        $order->update_status( 'completed' );
        $order->save();
    }
};

function bks_handle_custom_query_var( $query, $query_vars ) {
    if ( ! empty( $query_vars['delivery_status'] ) ) {
        $query['meta_query'][] = array(
            'key' => 'delivery_status',
            'value' => esc_attr( $query_vars['delivery_status'] ),
        );
    }

    return $query;
}

add_filter( 'woocommerce_order_data_store_cpt_get_orders_query', 'bks_handle_custom_query_var', 10, 2 );

This comes from the two different question which I answered on stackoverflow :

https://stackoverflow.com/questions/67838546/update-order-status-from-custom-statuses-to-completed-in-woocommerce-everyday-at/67844705#67844705

https://stackoverflow.com/questions/67689647/woocommerce-change-order-status-based-on-custom-meta-value/67755204#67755204

Deploying Sage(WordPress Starter theme) to your host using Github Actions

Bhanu Singh — Wed, 05 Aug 2020 05:40:38 GMT

I had recently started using sage for theme development of a new design upgrade for an old WordPress website. This website was hosted on Digital Ocean droplet having OpenLiteSpeed server installed on it.

If you look at the roots ecosystem, you will see that they Trellis project which take care of Production, Staging and Development environments. But in my case I couldn't change the host. To solve this problem I used Github Actions to setup continuous deployment.

#Steps Involved

Setting up Github Repository
Setting up your host machine
Command to push code to Host from local

Setting up Github Repository

I am considering that you have your sage code pushed to your remote github repository.

We will start by adding actions.

Click on the "Actions" tab in your repository.
Click on the button "Skip this and set up a workflow yourself ->" next. You will find it in the top section.
You will be taken to a page where which would be at your-repo/.github/workflows/main.yml. Change the name of the file to deployment.yml as we are creating actions to deploy our code.
Copy the code from below and paste it in your deployment.yml file.

name: Deployment

on:
  push:
    branches: [ production ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - uses: actions/setup-node@v1.1.0
      with:
        version: 12.x

    - name: Install dependencies
      run: |
        composer install -o
        yarn
    - name: Build
      run: yarn build

    - name: Sync
      env:
        dest: 'root@111.11.111.111:/var/www/html/wp-content/themes/theme-name'
      run: |
        echo "${{secrets.DEPLOY_KEY}}" > deploy_key
        chmod 600 ./deploy_key
        rsync -chav --delete \
          -e 'ssh -i ./deploy_key -o StrictHostKeyChecking=no' \
          --exclude /deploy_key \
          --exclude /.git/ \
          --exclude /.github/ \
          --exclude /node_modules/ \
          ./ ${{env.dest}}

In line no. 27 you will have to change the value of dest. This is what the value of dist will be for you @:. So for example bhanu@111.11.111.111:/var/www/html/wp-content/themes/theme-name
After this is done you have to on "Start Commit" button and merge it into your master branch.
You will have to add a secret DEPLOY_KEY which we will revisit after setting up our host.

Setting up your host machine

ssh into your machine. We will create a SSH key and add it our github repository in order for our give our github repository access to our host machine.

We do this in the following steps.

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Don't forget to change the email ID to yours. This will give you some prompts.

> Generating public/private rsa key pair.
> Enter a file in which to save the key (/home/you/.ssh/id_rsa): [Press enter]
> Enter passphrase (empty for no passphrase): [Type a passphrase]
> Enter same passphrase again: [Type passphrase again]

You have to press enter and skip all of the above. In some cases you may already have an SSH key and you can overwrite it in order to create a fresh one.

You have now created a new ssh at ~/.ssh/id_rsa.

###Adding SSH key to the ssh-agent

Enter the following command.

eval $(ssh-agent -s)
> Agent pid 59566
ssh-add ~/.ssh/id_rsa

Now we need to add our public key id_rsa.pub to the authorized_keys.

You can do this by.

# Getting inside SSH file
$ cd ~/.ssh/

# Copying pub key into your clipboard
$ cat id_rsa.pub
# This would print the content on the screen which you have to copy

# paste the content inside authorized_keys
$ nano authorized_keys
# To save press Ctrl+X -> y -> Press Enter

# Copy the content of public key
$ cat id_rsa
# This will paste the content of the public key which you have to copy starting from Start here till the end.

This is the point where we go back to point 7 of Setting up Github Repository

In your Github repository you go -> settings tab -> Secrets -> Click on New secretbutton -> Type the name of the secret as DEPLOY_KEY -> Paste the content of public key copied above here and hit Add secret.

We are almost done. We just have to test if everything is working fine or not.

Go to your local development and git push all the changes you want to deploy. When you are satisfied with the changes and you have pushed those in your master branch use this command.

git push origin master:production

This will push all the contents of master branch to production branch.

Now if you look closely at our deployment.yml file you can see these lines of code.

name: Deployment
on:
  push:
    branches: [ production ]

These lines tells that this action gets triggered when there is a push in the production branch.

You can look at your progress by going to the Actions tab. You would see an action is running. You can click on deploy to check the status and log of the same.

Let me know if you have any questions regarding this method, or you are stuck somewhere in the process.