With WordMentor, we set out to create an immersive learning experience that goes beyond traditional vocabulary drills. Our team of language enthusiasts and AI experts meticulously curated a collection of captivating words from popular authors.

Today, WordMentor is finally live on Product Hunt and we can’t wait for you to join us on this incredible journey. Visit our product page, share your thoughts, and be part of the discussion. Let’s elevate the way we learn words and inspire each other!

🔗 WordMentor on ProductHunt: https://www.producthunt.com/posts/wordmentor-3

Tasks are a great feature that allows you to use bash commands to create custom VSCode build commands. For example, you can create easy deploy, destroy, and synth commands for CDK.

To get started, you need to create the .vscode folder at the root of your project and add tasks.json in to that folder. In tasks.json, you can define a series of commands that VSCode needs to run in order to destroy, deploy, or synth your CDK code.

The code below shows the high-level structure of the task.json:

{ 
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [],
    "inputs": []
  }

In the tasks property, you can define as many commands as you wish to have. For example, the synth command is the easiest to create:

 {
    "label": "Synth CDK",
    "type": "shell",
    "command": "npx aws-cdk synth",
    "group": "build",
    "presentation": {
        "reveal": "always",
        "panel": "shared",
        "clear": true,
        "focus": true
    }
},

Before diving into creating a task for the deploy command, it's worth noting that I use aws-vault to manage my credentials and access AWS via the aws-cli. If you haven't installed it, I highly recommend doing so. You can use brew to install aws-vault:

brew install --cask aws-vault

Now, let's get back to creating the configuration for the CDK deploy command. As we have multiple profiles and accounts set up in aws-vault, VSCode must know which account we wish to deploy our code to. To achieve this, I will leave a placeholder in the configuration for the profile name:

{
    "label": "Deploy CDK",
    "type": "shell",
    "command": "aws-vault exec --prompt=osascript ${input:accounts} -- npx aws-cdk deploy",
    "group": "build",
    "presentation": {
        "reveal": "always",
        "panel": "shared",
        "clear": true,
        "focus": true
    }
}

The ${input:accounts} is the placeholder. But, we don't want to type the profile's name. Instead, we want VSCode to show us the list of the available profiles so we can choose from them. To achieve this, we can define an input in the input section of the tasks.json to list those profiles for us.

However, VSCode's inputs do not run bash scripts. To fix this, you need to install Tasks Shell Input from the marketplace.

{
    "id": "accounts",
    "type": "command",
    "command": "shellCommand.execute",
    "args": {
        "command": "aws configure list-profiles"
    }
}

We can also use the same approach for the CDK destroy command as well. The whole file should looks like this:

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Deploy CDK",
            "type": "shell",
            "command": "aws-vault exec --prompt=osascript ${input:accounts} -- npx aws-cdk deploy",
            "group": "build",
            "presentation": {
                "reveal": "always",
                "panel": "shared",
                "clear": true,
                "focus": true
            }
        },
        {
            "label": "Destroy CDK",
            "type": "shell",
            "command": "aws-vault exec --prompt=osascript ${input:accounts} -- npx aws-cdk destroy",
            "group": "build",
            "presentation": {
                "reveal": "always",
                "panel": "shared",
                "clear": true,
                "focus": true
            }
        },
        {
            "label": "Synth CDK",
            "type": "shell",
            "command": "npx aws-cdk synth",
            "group": "build",
            "presentation": {
                "reveal": "always",
                "panel": "shared",
                "clear": true,
                "focus": true
            }
        },
    ],
    "inputs": [
        {
            "id": "accounts",
            "type": "command",
            "command": "shellCommand.execute",
            "args": {
                "command": "aws configure list-profiles"
            }
        }
    ]
}

When you install a new application on your mac, the setup will throw a lot of files through your system directories. Even after the installation, many apps will continue to create config and helper files in different directories.

Since MacAppStore came, lots of applications are now available on the MacAppStore, and uninstalling them is just a push of a button, but still, there are tons of applications still out there that are not available on the MacAppStore for different reasons—like requiring access that is not permitted by the MacAppStore guidelines.

Many application like CleanMyMac has come to solve the issue of uninstalling outside MacAppStore applications and taking care of completely cleaning their configs, helper, logger and etc files. Those applications are expensive for such a simple task and also not all of them are that good at finding extra files. In this blog post, I am going to show you how I used a single-line bash script to fully uninstall any application that I don’t like to have anymore.

Understanding macOS directories

macOS uses specific directories to store user, system, and application configurations. It is crucial to understand these libraries as it will help us to differentiate the application files from the other files.

macOS uses plist files as configuration files. So, it is the most probable file type you will see any application creates. Also, apple likes to store these configurations under two directories. The first directory that is holds applications and system-related stuff is /Library this folder holds files that are owned by the root and are available for all the users. In contrast, the ~/Library holds user-level files.

The bash commands

To begin with, we need to understand three commands in bash first, and then by the power of bash pipe we will put them together to create our own easy uninstaller.

1. The mdfind command

The first command to understand it is mdfind. This command is responsible for searching for files through the whole system. By passing the option command -name we can look for a regex pattern or an exact word in files names and paths.

Here is an example of running the mdfind command to find all files that have “iterm2” in their name:

mdfind -name iterm2

Here is an output example:

/Users/kiarash/Library/Application Support/iTerm2/iterm2-daemon-1.socket.lock
/Users/kiarash/Library/HTTPStorages/com.googlecode.iterm2
/Users/kiarash/Library/Application Support/iTerm2
/usr/local/Caskroom/iterm2
/usr/local/Cellar/ncurses/6.3/share/terminfo/69/iterm2-direct
/usr/local/Cellar/ncurses/6.3/share/terminfo/69/iTerm2.app
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask/Casks/iterm2.rb

2. The vipe command

As I mentioned before many uninstalling applications aren’t able to fully clean the extra files because they only look for specific files in specific directories that I mentioned earlier. What I like to do instead is to use mdfind to search for any related files and then use an editor like neovim to list all the candidate files and then edit the list of the candidates and return it back to the bash script to be removed.

💡Note 1: If you haven't used neovim but vim instead, don't worry! neovim is just the community version for vim.

💡Note 2: If you haven't used either neovim or vim check out this tutorial or use nano instead.

To achieve that goal I used vipe. This command basically takes the output of the first command, passes it to the default editor–which here is neovim–and then after saving and quitting from the default editor, it will pass the edited and saved list to the second command.

Before using the vipe command, we need to install it via brew as it is not installed by default in macOS. To install vipe we need to install a formula named moreutils which contains vipe.

brew install moreutils

Now you can use the pattern below to use the pass, edit, and pass process between two commands:

command1 | vipe | command2

3. The xargs command

Let’s say we successfully listed all the related files to the application which we wish to be removed completely. It is obvious now that we need rm command to delete them, but each file is on a separate line. Also, the rm does not accept the pipe output as argument. To address this issue we will use the xargs command. This command can take a pipe output and run a bash command with that. Plus, the -L option will split the lines and loop through them i.e. it will rm command separately with each line. The script below demonstrates how we can remove a list of files from a text file as a source:

cat source.txt | xargs -L 1 -I {} rm -r {}

4. Put it all together

Now that we learn about each command, it's time to combine them together and make a script out of them. Save the script below in the unapp.sh file.

#!/bin/zsh
mdfind -name $1 | vipe | xargs -L 1 -I {} rm -rf {}

Make the script executable by:

sudo chmod +x unapp.sh

No we can uninstall any app fully using the command below:

./unapp.sh iTerm2

References

• moreutils documentations
• xargs
• NeoVim home page

SageMaker is one of the central services for data scientists. SageMaker took much attention over the last few years. While many data scientists are trying to learn more about it, developers of SageMaker are constantly working on new solutions and abilities to bring it to the SageMaker. SageMaker provides tools for both the production and experimentations environments.

When it comes to production, SageMaker benefits from the power of dockerization to create isolated scalable computation machines and workflow. If you are not familiar with these concepts, you can learn them from AWS Documentation for training and processings data jobs.

Passing data to these containers is one of the challenging steps of productionizing an ML application. In this blog post, I am going to break down into detail all the paths and possible ways to send or receive data to a training or processing job container.

Methods of loading and saving datasets in SageMaker

S3 is one of the most popular AWS services that is being used for storing data and integrated with many other AWS services. SageMaker is not an exception and has implemented native solutions to load and save data into S3.

Normally, you might run two types of algorithms on SageMaker: a processing job and a training job. You can use this documentation to see how to load data from S3 to a SageMaker processing job, and this documentation for a training job. The code below demonstrates how to define an input source for a processing job:

from sagemaker.processing import ProcessingInput

ProcessingInput(
     source="s3://sagemaker-sample-data/processing/census/census-income.csv", 
     destination="/opt/ml/processing/input"
)

There are two options for loading data from S3 in SageMaker, File mode, and Pipe mode. File mode will copy the input files from an S3 bucket into the container. This is the default method and is the popular method of communicating with S3. In the code above, Since we did not pass the S3InputMode, the file mode is considered as the method of loading data. Also, there is a pipe mode that streams the data instead of copying it and, as a result, will not use storage inside the container.

There is one more well-known method of communication with S3 which we will cover later in this post but first, let us talk about the destination paths of those inputs inside the containers.

Understanding Container's paths

When we are in SageMaker training or processing container, we are working with directories under the path /opt/ml. The SageMaker itself also uses this directory as the root directory to save and load all files related to the training/processing algorithm. For instance, hyperparameters are stored as a JSON dictionary in /opt/ml/input/config/hyperparameters.json.

There are three states you wish to transfer data from S3 to a container or vice versa. Based on that, we can classify the paths in a container:

1. Input Data: The very raw data you wish to feed to your preprocessing step must be saved to /opt/ml/processing/input/.
2. Processed Data: The output of your preprocessing step must be saved under the processing directory under different channels of your choice. For instance, /opt/ml/processing/training, /opt/ml/processing/test, and /opt/ml/processing/validation.
3. Trained Model: The artifacts of your training model from the training step must be saved to /opt/ml/model.
4. Training Output: The extra data when training a model must be saved to /opt/ml/output/ and /opt/ml/output/data.

Using S3 SDK, Boto3

Boto3 is the most available solution to download or upload data to AWS S3. It is easily accessible inside the containers. I usually won't go with this option as it is not completely SageMaker friendly, and there is also a better native way to load data from S3 into the container. However, there are some cases in which I prefer to use Boto3 alongside the SageMaker native options to load data, like loading a big configuration. Nevertheless, the code below demonstrates how to download or upload data from S3 using Boto3

import boto3

s3_client = boto3.client('s3')

# Upload the file to S3
s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')

# Download the file from S3
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')

References

• SageMaker Python SDK
• SageMaker Documentations

Graphs have taken a lot of attention during the last years, from graph machine learning methods, including Graph Neural Networks, to Graph Databases. Even on Medium, people are posting graph articles more than before. One thing that I believe most data scientists underestimates is the power of graphs in visualization and storytelling.

Many see graphs as a complex and expensive solution for modeling data science problems. However, they still provide a great way to visualize data that no other chart or visualization can.

In this blog post, I will use the Mathematicians of Wikipedia dataset to investigate relations between mathematicians and their advisors using NetworkX to analyze and visualize.

What is NetworkX?

NetworkX is one of the most popular frameworks to work with. I think it is the scikit-learn of graph world! It is most probable to find NetworkX sooner than any other graph library when you start to learn about graphs and graph frameworks.

NetworkX provides many graph implementations, algorithms, and methods of analysis. Even though it is slow for many real applications, it still offers great functionalities worth learning.

Understaning the data

I have found the dataset from Kaggle. The table below demonstrates all the available features for this dataset that describes mathematicians. Note that the examples do not come from the same row.

The highlighted column names (mathematicians and doctoral advisor) are the columns in which I am interested in analyzing and showing how mathematicians are connected. So then, my first step is to load the dataset and clean it.

import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np

df = pd.read_csv('../data/data_cleaned.csv')

The first problem with the data is that many of its columns, including doctoral advisor, are a list of names embedded as a string. As I am only interested in the doctoral advisor column, I will only fix this column, but you can add the column names to the list_type_columns to fix it as well.

list_type_columns = [
    'doctoral advisor'
]

df[list_type_columns] = df[list_type_columns].fillna('[]')

for column in list_type_columns:
    df[column] = df[column].str.replace("'", '', regex=False)
    df[column] = df[column].str.replace("[", '', regex=False)
    df[column] = df[column].str.replace("]", '', regex=False)
    df[column] = df[column].str.split(',')

    df = df.explode(column)

Now that I have handled my dataset issues, including Null values, by replacing them with empty lists and fixing the string embedded list to real lists, I can save it to a CSV file to use later, so I won't need to process my dataset every time.

df.dropna(inplace=True)
df[['mathematicians', 'doctoral advisor']].to_csv('../data/adv.csv', index=False)

I only saved the columns I needed, including mathematicians and doctoral advisor.

PageRank Algorithm

PageRank computes the rank of each node based on the number of incoming edges. Google initially developed PageRank to calculate the importance of web pages.

PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites.

Using RankPage in NetworkX is really easy:

G = nx.DiGraph(nx.path_graph(4))
pr = nx.pagerank(G, alpha=0.9)

Graph Visualization has more insights

Now that we have loaded our dataset, cleaned it, and fixed its issues, let's see the top mathematicians with the most students.

top_ten_advisor = df['doctoral advisor'].value_counts().sort_values(ascending=False).head(15)
fig, ax = plt.subplots()
ax.barh(top_ten_advisor.index, top_ten_advisor.values)
ax.set_xlabel('Number of Students')
ax.axvline(x=13, c='red')
plt.show()

output:

Even though the above image shows the most famous mathematicians and their students, it does not describe how they are connected and leaves lots of questions unanswered:

1. Are these top advisors connected by a shared student?

2. Are these top advisors also advisors of each other?

3. Do students also advise other students?

To address the questions above, a graph representation can help us. First, let's drop all the advisors with less than 13 students.

minimum_count = 13 
df = df[df['doctoral advisor'].isin(df['doctoral advisor'].value_counts()[df['doctoral advisor'].value_counts() >= minimum_count].index)]

We create our graph in NetworkX and calculate the PageRanks ratings.

graph = nx.DiGraph()
graph.add_nodes_from(np.unique(df.values.flatten()))
graph.add_edges_from(df.values)

pr = nx.pagerank(graph)

names, ranks = zip(*pr.items())
pr_df = pd.DataFrame(data={'mathematicians': names, 'rank': ranks})
pr_df

Now that we have both graphs and ranks, we can draw.

fig = plt.figure(1, figsize=(30, 20), dpi=100)

pos = nx.spring_layout(graph, k=1.1*1/np.sqrt(len(graph.nodes())), iterations=20)
nx.draw(graph, node_size=pr_df['rank'].values*10000, with_labels=True, pos=pos, edge_color='gray')
plt.show()

References

• Mathematicians of Wikipedia

• NetworkX Documentations

• PageRank

There are many scenarios in which bash scripts can be used to automate processes. Sometimes, I like to test things in Python. For instance, sometimes one of my colleagues asks me if something is possible in Python or how it is done. In those cases, I don't want to create a new project, but rather to create a new folder, throw a bunch of python files in, and figure out things. After some time, these test projects make a big mess in my directories, which I don't like.

To fix this, I made a bash script to remove all the files and folders that I have created under "temp" every time I boot my Mac. This blog post will describe how I managed to use macOS launchd to schedule a bash script to delete all the test files and folders.

What is launchd?

As a Linux user or at least someone who has worked with Linux, you may be familiar with systemd. launchd is exactly the same, but it can do more like a job scheduler (corn replacement) and it is much more reliable than systemd.

1. How to interface launchd?

Anyhow, to interact with Launchd, use launchctl in your bash shell. Same as the systemd and the systemctl commands, isn't it? You can see more information about launchctl by typing man launchctl in your terminal window.

2. What launchd configuration files look like?

Having learned about the concept of launchd and its interface launchctl, it is now time to understand how we can define a new configuration for this. In contrast to systemd, which uses TOML config files, launchd makes use of plist files to define a configuration. Below is an example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.hello</string>
    <key>ProgramArguments</key>
    <array>
        <string>hello</string>
        <string>world</string>
    </array>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>

3. Where to save launchd configuration files?

There are three locations in which you can store a launchd configuration file. If your application is supposed to run as a daemon service, you can store it under /System/Library/LaunchDaemons/. If your application is supposed to run whenever a user logs in, you should store it under /Library/LaunchAgents. Finally, If your application is supposed to run whenever a specific user logged in it must be stored under ~/Library/LaunchAgents.

Creating the automation

It's time to set up the automation now that we know how launchd works. Let's create the bash script file first:

rm -rf $HOME/projects/temp && mkdir $HOME/projects/temp

You need to save this script to ~/.scripts/loginscript.sh. The script will delete the files and folders in the temp folder every time you run it. We will now need to create the config file for Launchd so the script runs every time we log in.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.user.loginscript</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/zsh</string>
        <string>PATH-TO-HOME-DIRECTORY/.scripts/loginscript.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Let's save this to ~/Library/LaunchAgents/com.user.loginscript.plist. Before registering it to launchd we first need to get the user id as it needs by launchd:

id -u

Copy the output and run the command below to register the configuration to launchd:

sudo launchctl bootstrap gui/<USER-ID> ~/Library/LaunchAgents/com.user.loginscript.plist

References

• Apple Documentions for Launchd

There are many times that you would prefer a custom docker image over what Sagemaker provides for you. People are using some tricks to install custom packages of their needs inside those default containers, but that is not clean nor reliable to do. Also, building a Docker image and pushing it to Elastic Container Registry is another kind of a problem. It has lots of steps and lots of commands to memorize to push the image to ECR every time you wanted to change something, adding a feature, or fixing a bug.

Amazon Web Services provides lots of tools to create and build infrastructures that made your life easier. With a combination of a git server (like Github), CodePipeline and CodeBuild you can automate the process of building a custom docker image and push it to ECR, and now you can get all of your focus on designing the model 🍻!

In this post I’m going to use GitHub as my git repository server but feel free to replace it with Bitbucket or Amazon CodeCommit service. The structure of the integration of these three systems is pictured below.

Elastic Container Registry

ECR is a private docker repository where one can push and pull its images to it. The ECR also has a public repository where anyone can download images with no authentication required. The policy access is available under permissions of a private repository which enables us to grant specific AWS accounts access or define wildcard access (which means anyone can have access) for pulling images.

To create a new ECR repository:

1. Go to the AWS console

1. Click on Create repository

1. Click on the view push commands to see commands to push to the repository of your interest.

CodePipeline

CodePipeLine act as an entry point of the structure. By pointing a new pipeline to a git repository and selecting a branch, the pipeline will subscribe to commits and code updates. Every time you push a new commit to the chosen branch of this repository, a web hook trigged the pipeline to pull changes and begin the process of building and pushing to ECR.

To begin with, you need to create a new pipeline from AWS console. Follow the config provided below and leave other configs with their default values.

Step 1

• PipeLine name: logical name referred to the name of our project.
• Service Role: choose New service role, so AWS create least privileged role for running pipeline.
• Role Name: the name of the role that AWS will create on behalf of you.

Step 2

• Source provider: choose the git repository service to connect to from the provided list.
• Repository name: your source repository.
• Branch name: branch of the repository to pair with.

CodeBuild

Step 3

In this step you can choose to connect to a created CodeBuild service or create a new one. In a CodeBuild service we define bash instructions to build and push our docker image from pulled source by CodePipeline service. The CodePipeline service is responsible for starting build process by calling the CodeBuild service.

Since we are creating new structure, we need to create new CodeBuild project for this repository by hitting the Create project button.

🚨 Region must be the same for all three service we are using in this architecture.

• Project name: logical name referred to the name of our project.
• Operating system: select Amazon Linux 2
• Runtime(s): select *standard*
• Image version: select Always use the latest for this runtime version
• Privileged: check the check box so we could be able to push to ECR.
• Service Role: choose New service role, so AWS create least privileged role for running pipeline.
• Role Name: the name of the role that AWS will create on behalf of you.

The buildspec.yml file needs to be created, so CodeBuild can automatically build and push the image. To do this, simply replace the following commands in the scripts with the ones seen in ECR-view push commands:

version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR and Docker Hub
  build:
    commands:
      - echo Build started on `date`
      - echo Builing image from docker file.
      - docker build -t <YOUR REPOSITOR YNAME>:latest -f Dockerfile .
      - echo Pushing image to Amazon ECR.
      - aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin <AWS ACCOUNT ID>.dkr.ecr.eu-west-1.amazonaws.com
      - docker tag <YOUR REPOSITOR YNAME>:latest <YOUR ECR REPOSITORY PATH>:latest
      - docker push <YOUR ECR REPOSITORY PATH>:latest
  post_build:
    commands:
      - echo Build completed on `date`

Summary

In a nutshell, you will be able to autonomously generate machine learning model images with this architecture. This automation will bring you more stability of the codes and infrastructure because we eliminate the process of human interventions from pushing the image to the ECR. CodeBuild service builds and pushes the images from the ECR when we push a commit to the master branch. The trigger for this process occurs when commits are pushed to the master branch.